Code Representation Pre-training with Complements from Program Executions

W1: authors seem to be completely unaware of the vast literature of neural code models based on dynamic executions
ANS: Thanks for suggesting these works on training neural networks by program executions. We have added discussions about these works in our paper. Indeed, [1,2,5] explore execution traces for code embedding learning and [3,4] leverage input-output pairs for program synthesis. In comparison to them, we demonstrate the effectiveness of learning discriminative code embeddings from input-output pairs (test cases) for code analysis/understanding. Test cases are an essential subset of execution traces, which are easier to obtain [3]. More importantly, by exploiting test cases, we can take advantage of fuzzing which is almost a language-agnostic process to vastly reduce the complexity of data collection and make it easy to be applied to different programming languages to build code understanding models with multilingual support.

W2: The evaluation task of clone detection is poorly handled. incorporating execution traces are semantic information that totally ignores the syntactic feature
ANS: We appreciate the comment, but there seems to be a misunderstanding. We didn't involve any execution information when fine-tuning on the clone detection task. In fact, one of the key contributions of our work is to ensure that only the source code (or its syntactic representations) is needed for downstream code understanding tasks (including clone detection) while still yielding competitive performances when compared to Zhao et al's work [6]. Specifically, Zhao et al explicitly use test cases for tuning/testing in the clone detection downstream task and achieve 92.01% mAP on POJ-104 while our method achieves 93.0% without incorporating execution traces as semantic information when performing clone detection, both adopting the CodeBERT as the base model. We have reiterated this at the beginning of our experiment section to make this more explicit.

W3: Fig 3 demonstrates that in Defect DIM and DID is not necessary because removing them actually yields a bigger gain over FuzzPretrain
ANS: Applying either DID or DIM solely may outperform our FuzzPretrain occasionally to a very limited extent. We hypothesise this is due to our supervision signal from joining DIM and DID for global code modelling in sequence-level is sometimes overstrong. This is likely to dominate our continual learning of the base models which were pre-trained with token-wise objectives to be sensitive to local fine-grained features revealing defects. However, we would like to argue that the general performance advantages yielded by FuzzPretrain when being applied to different base models on different tasks demonstrate not only its effectiveness but also its generality, which is critical for a pre-training method. It is also the reason why pre-trained models should be tested on several different tasks to demonstrate their effectiveness.

References
[1] Ke Wang, Rishabh Singh, and Zhendong Su. Dynamic neural program embedding for program repair. arXiv preprint arXiv:1711.07163, 2017
[2] Ke Wang and Zhendong Su. Blended, precise semantic program embeddings. In Proceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation, pp. 121–134, 2020
[3] Eui Chul Shin, Illia Polosukhin, and Dawn Song. Improving neural program synthesis with inferred execution traces. Advances in Neural Information Processing Systems, 31, 2018
[4] Xinyun Chen, Dawn Song, and Yuandong Tian. Latent execution for neural program synthesis beyond domain-specific languages. Advances in Neural Information Processing Systems, 34: 22196–22208, 2021
[5] Jordan Henkel, Shuvendu K Lahiri, Ben Liblit, and Thomas Reps. Code vectors: Understanding programs through embedded abstracted symbolic traces. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 163–174, 2018
[6] Jianyu Zhao, Yuyang Rong, Yiwen Guo, Yifeng He, and Hao Chen. Understanding programs by exploiting (fuzzing) test cases. ACL, 2023

W1: there is work that does similar things

• they have not cited "TRACED..." leverages dynamic information, specifically executable inputs and corresponding execution traces, for pre-training
  ANS: Thanks for suggesting this contemporary work. We have discussed this work in our updated paper. Here, we'd like to clarify our difference is that we explored test cases as another modality of data in inputs while they use execution traces to build labels for learning their model's outputs. We believe these are two orthogonal attempts of a similar idea and are not contradictory. Specifically, our FuzzPretrain showed superior performance on clone detection in POJ-104 to TRACED by synthesising comprehensive test cases to provide a holistic picture of the functionality of code snippets, while they are better on defect detection by explicitly learning from the finer-grained internal status in execution. Since they are orthogonal, the two methods can possibly to combined to achieve even better results. Some initial comparisons and discussions have been added to the related work section and Table 3, respectively.

• "Understanding Programs by Exploiting...", though they are not actually using them for pretraining
  ANS: We would like to stress that, as discussed in Section 2 in our paper, we dedicatedly designed our model to avoid requiring additional information (i.e., test cases) about code in downstream tasks, and this tells our difference to the suggested work. Test cases are only used in the unsupervised pre-training phase and never used in downstream tasks in experiments in this paper. We have revised the paper to make this point clearer.

W2: they present results for an approach that does use dynamic information for pretraining: CodeExecutor...they do not present this on the main code search task in Table 1. many of the "state-of-the-art" models listed in Table 3 are not included in Table 1 for code search. The same goes for the analyses in Figures 4-5
ANS: This is because our results in Table 1 and Figures 4-5 are NOT for comparisons to the state-of-the-art but to provide in-depth analyses of our model for its comprehension. Therefore, we included only the models that trained on the same data as our FuzzPretrain to conduct fair comparisons and derive reliable conclusions, in addition to demonstrating our competitiveness to the state-of-the-art that trained on different data in a separate table (Table 3). We further evaluated CodeExecutor on the code search task in Table 1 without re-ranking by execution traces. It yielded 3.54% overall mAP vs 16.1% and 30.2% by our FuzzCodeBERT and FuzzUniXcoder, respectively. These results imply that the CodeExecutor seems to overfit to the execution tasks while not being able to produce discriminative feature representation for source code directly. This is consistent with the results we reported in Table 3.

W3: the underlying model that is used, UniXcoder, was originally designed to also handle generative tasks. The authors do not report results for generative tasks
ANS: We evaluated UniXcoder with and without FuzzPretrain on the python dev split of the line-level code completion task from CodeXGLUE. For fair comparisons, we also report the results of our MLM baseline which was trained on the same CodeNet data as ours without using the generated test cases. As shown in the below table (results in which have also been added to the appendix of the paper), the benefits brought by FuzzPretrain are small but consistent. This is because our designs for dynamic information modelling are all about learning holistic representation of complete code snippets to be discriminative, regarding their functionality but not token-level generation. This contributes the most to code understanding and does not directly benefit code generation. However, we believe that, as a seminal work of incorporating test cases into programming language model pre-training, our work will inspire future methods which could improve code generation by taking advantage of test cases in pre-training.

Q1: Why use test cases to represent code dynamic information, can test cases are sufficient to represent code dynamic information?
ANS: Test cases provide explicit indications of programs' functionality without any distractions from implementation variations, e.g. the data-flow graph or execution trace of the same functionality might vary depending on its implementation. They reveal programs' execution behaviours and are sufficiently informative to support challenging code-related tasks like code generation [1,2]. More importantly, test cases are easier to collect and thus more scalable for different programming languages to enable multilingual code understanding.
We added the comparisons to a contemporary method [3] which was trained using execution traces. It turns out that our FuzzPretrain is superior on the clone detection task which evaluates the code feature's sensitivity to syntax variations of the same functionalities, i.e. our FuzzCodeBERT yielded 93.0% mAP vs 91.2 by TRACED [3]. Detailed results and discussions about this can be found in the revised Table 3 in our paper.

Q2: How to construct negative samples in DID pre-training task at equation 3? Please give more explanation.
ANS: The negative sample set is constructed randomly. As illustrated in Section 3.3, we simply followed He et al [4] to feed all the samples in a mini-batch into a queue and remove the oldest one inside when the queue is full in each forward pass. We then take the queue as the negative sample set in the next iteration. Given that our mini-batch is constructed randomly, the negative sample set is also random in every iteration. More sophisticated methods may lead to even better results.

Q3: Can FuzzPretrain wok well in some code generation tasks such as code summarisation, code completion?
ANS: As stated in Appendix A.3 of our updated manuscript, we tested but didn't observe notable improvements brought by FuzzPretrain to the base models on generative tasks. To be concrete, we yielded 42.73%/72.03% Exact Match/Edit Sim on the python dev split of the line-level code completion task in CodeXGLUE vs 42.68/71.88 by UniXcoder. We hypothesise this is due to all of our designs for dynamic information modelling encouraging holistic comprehensions and discrimination of code snippets but not on token-wise generation.

Dear Reviewer EByC,

We are truly grateful for your review and hope that our responses have effectively addressed your concerns. As the final deadline for reviewer-author discussions draws near, we are eager to receive any additional feedback you may have. We are more than willing to offer further clarification and engage in a deeper conversation if any of your concerns remain. Many thanks!

Best regards,
Authors

Q1: In table 1, why is the first row in grey? Same question with table 2.
ANS: The first rows in Tables 1&2 are marked in grey because they are not directly comparable to the other rows in the same tables as training on different data. They were reported to demonstrate that we are able to obtain substantial improvements over our starting points even though our continual training was conducted on a smaller and less diverse code corpus.

Q2: For table 1, what does the languages in the first and second row mean? what about C/C++
ANS: The first and second rows in the header indicate the programming languages of the query and the candidate code snippets in the code-to-code retrieval (search) task, respectively. We followed the setting of this task introduced in UniXcoder to evaluate on the three reported languages. We elaborated on this in the revised caption of Table 1. Besides, we further evaluated on 50 randomly held-out coding problems in the C++1400 split of CodeNet. Our FuzzCodeBERT yielded 68.65% mAP while CodeBERT and CodeBERT-MLM achieved 13.22% and 24.08%, respectively.

Q3: Still table 1, is "w/o DIM" and "w/o DID" rows accumulative? If not, is "w/o DIM + w/o DID" the same as MLM variants in the table?
ANS: No, they are not accumulative. And yes, "w/o DIM + w/o DID" is the same as the MLM variant. Table 1 has been refined to clarify this.

Q4: Can you explain why you alternate the three objectives between different mini-batches, and not simply add them or interpolate between them?
ANS: We alternated between different objectives for the concerns of efficiency and GPU consumption. The three training objectives require inputs in different forms, hence, three forward passes are needed for their joint optimisation. This will lead to accumulated GPU consumption and refrain us from fitting the desired batch size into GPUs. Such an alternative training strategy has also been used in previous works like UniXcoder and GraphCodeBERT.

Q5: Since the model is both trained and tested on CodeNet, can you elaborate on how you split the data and if some precautions are taken to prevent implicit data leakage?
ANS: We adopted the testing set provided by UniXcoder and the benchmark datasets provided in CodeNet for training. Given that the testing set involves more than 70% of the coding problems in CodeNet, it is challenging to exclude them from training data. Therefore, we consider the searching of those overlapping samples as transductive inference problems. This is also practical given that the training data of the latest code models cover a large proportion of the open-source projects in Github and is likely to involve the code-of-interests to users.
As suggested, we have further evaluated in an inductive setup where the query and the candidate code snippets are submissions to 50 random coding problems of each programming language that have never been seen during pre-training. The header in the below table indicates the data splits provided by CodeNet. As can be seen, the superiority of our FuzzPretrain holds.
That is, these results show that our model is effective not only in the transductive inference setup for code search, but also in an inductive setup where no training/test overlap exists. These discussions have been added to the Appendix A.2 of the paper.

Dear Reviewer LioK,

We are truly grateful for your review and hope that our responses have effectively addressed your concerns. As the final deadline for reviewer-author discussions draws near, we are eager to receive any additional feedback you may have. We are more than willing to offer further clarification and engage in a deeper conversation if any of your concerns remain. Many thanks.

Best regards,
Authors