SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Models

The main weakness to me is that all the conclusions and insights are somehow predictable.

We addressed this concern in the second paragraph of General response. We hope our response covered the concerns. Please let us know if we have to provide more information about that.

Could you conduct an ablation study to show the performance gain versus different amounts of extra data? This would give more valuable clues to the community as we would know using how much data for fine-tuning would be the best tradeoff.

Thank you for the insightful suggestion. We are working on the suggested experiments and try our best to provide the results before the end of the discussion period.

Can you conduct an ablation study of increasing the amount of fine-tuning data and the fine-tuning time? This might decrease the advantage of LoRa against FT.

Thanks for the suggestion. We provide the fine-tuning time of above experiments for different extra data (E.g., presenting the fine-tuning time of adding an extra amount of masked data). Please let us know if the question mainly asked for comparing LoRA and FT's running time with different numbers of data (e.g., 1k, 10k, 100k).

I noticed that the authors consistently misuse \citet and \citep throughout the whole paper. I suggest the authors proofread the paper a few more times to improve the readability.

Thank you for bringing this to our attention. We had a pass over the paper and resolved the issue with \citet and \citep. The new version of the paper is uploaded.

Here, we provide the results for the following question. We hope that the description and the results resolve your concerns.

Could you conduct an ablation study to show the performance gain versus different amounts of extra data?

In Table-1 and Table-2 we show the effect of revealing different percentages of the masked data on the model's performance. Specifically, we showcase CodeT5+ performance in different scenarios by revealing 25%, 50%, and 75% of the masked data (The data was masked for the OOD scenarios). Table-1 presents results for the text-to-code task, while Table-2 displays results for the code refinement task.

Table-1 and Table-2 demonstrate that the model can gain a high performance even by revealing 25% (~ 0.75% of training data). For instance, in Table-1, within length extrapolation scenarios, the full fine-tuned model notably showed relative performance increases from 5.0% (OOD) to 64.63% (Few-25%). Furthermore, both tables indicate that revealing 50% and 75% of the masked data can enhance the model's performance across different scenarios. Nevertheless, the observed performance gains for Few-75% are less apparent compared to the Few-50% and Few-25% cases.

Table-1: Overall model performance results for different scenarios with different amounts of data in text-to-code task. Few-XX% show the results of revealing 25%, 50%, and 75% of the masked data to the model. Note that OOD and Few-50% were presented in the paper.

Table-2: Overall model performance results for different scenarios with different amounts of data in code refinement task. Few-XX% show the results of revealing 25%, 50%, and 75% of the masked data to the model. Note that OOD and Few-50% were presented in the paper.

We try our best to provide the requested ablation study. Please let us know if we need to provide any other experimental results. We appreciate a short reply to let us know your thoughts. Thanks a lot for your time.

Can you conduct an ablation study of increasing the amount of fine-tuning data and the fine-tuning time?

In Table-1, we present CodeT5+ fine-tuning times for various data quantities, specifically focusing on the fine-tuning time for both full fine-tuning and LoRA fine-tuning over 10 epochs. The results showcase the fine-tuning time using 1k, 10k, and 100k data. Overall, Table-1 indicates that full fine-tuning exhibits a slower fine-tuning time compared to LoRA fine-tuning across varying amounts of data.

Table-1: Fine-tuning time (in second) of CodeT5+ model for text-to-code task using different number of data. FT refers to full fine-tuning method, and LoRA denotes LoRA fine-tuning model.

Thank you for the time taken to review our work. We hope we can address your concerns.

The paper mainly tests several existing methods for the forgetting task of code datasets. The novelty of this work is not clear from the conclusion and experiments now.

Note that the main focus of the paper is not forgetting issues. We are mainly studying the OOD issues of the pre-trained source code models, where forgetting issues can be a subset of OOD issues in our scenarios. For example, a model can perform poorly in the semantic scenario when it does not see similar examples during pre-training or fine-tuning.

About the novelty of the work: to the best of our knowledge, this is the first work that studies the OOD behavior of large language models (e.g., Code Llama and CodeT5+) in generation tasks. To do that, we propose a systematic approach and, for the first time, define OOD scenarios for source code data along three code-related diverse dimensions: length, syntax, and semantic. Our results along each dimension raise awareness about the potential OOD issues of full fine-tuned and LoRA fine-tuned source code models.

As an experimental paper, it lacks enough comparison of methods that prevent forgetting, e.g.Wise-FT.

Thanks a lot for your suggestion. We agree that there are different methods to mitigate catastrophic forgetting. However, in this work, our main focus is to show the performance of two widely used fine-tuning approaches in the source code OOD scenarios. We show that regardless of architectures and parameter sizes, the models' performance dropped in various OOD scenarios (Even for Code Llama 6B, which was fine-tuned using the LoRA method). The paper's main message is to raise awareness about the OOD issues of the fine-tuned LLMs in the source code domain.

For analyzing the forgetting phenomenon, there lacks the analysis on different hyperparameters' choice of finetuning or the size of the dataset.

In our study, for a fair comparison with the baselines, we use the hyperparameters' of previous works (We specified these hyperparameters in the paper and our Reproducibility Statement), and we also used two well-known datasets with different sizes. As mentioned above, our main goal was to investigate the performance of two fine-tuning methods in OOD scenarios. However, we think analyzing different hyperparameters' choices would be an interesting future study.

The findings from the code data do not appear to be surprising and align with similar observations in prior studies involving OOD scenarios with image data.

We addressed this concern in the second paragraph of General response. We hope our response covered the concerns. Please let us know if we have to provide more information about that.

Do the three types of code OOD scenarios accurately represent realistic code OOD data?

We also address this question in the second part of General response (Paragraph 6).

The positive impact of a small amount of labeled data on generalization is in line with expectations, given the motivations within the realm of few-shot learning.

We agree that the output of using a few examples might be intuitive. However, before this study, we did not know how the models in each OOD scenario could benefit from the small amount of data. For example, in Table-1 we show that adding a small amount of relevant data to fine-tuning data for CodeT5 can lead to a gain of ~100% of relative performance. Therefore, our diverse scenarios and the few-data regime analysis provide insight toward shaping future datasets.

The experimental details have been mentioned several times in various paragraphs. I would kindly recommend refining the presentation to make it more concise and informative.

Thank you for bringing this to our attention. We will have another pass over the paper and try to remove any repetitive information before the end of the discussion period. It would be great help if you can point out the specific part of the paper that was repetitive.

The finding that fine-tuning may distort the pre-trained features, cause catastrophic forgetting has already been studied in various NLP and CV tasks

We addressed this concern in the second paragraph of General response. We hope our response covered the concerns. Please let us know if we have to provide more information about that.

This paper also mentions the works of OOD analysis in programming languages in RELATED WORKS, (a) their relationships with this paper should be discussed in more detail. (b) Why not use their pre-defined scenarios for evaluation?

a- Thanks a lot for pointing this out. We clarify the relationship of Bui and Yu [1] with our work in the paper (in blue). You can find it in the newly uploaded version.

b- Bui and Yu [1] only focused on the OOD issues in the code classification tasks, which are, in general, easier to define (e.g., removing the data that belongs to a specific label). However, in our work, we are defining OOD scenarios for code generation tasks, which are more widely used in various applications.

There are many methods proposed in CV and NLP to mitigate catastrophic forgetting in fine-tuning, but only the LoRA method is evaluated, which makes the experimental results not comprehensive enough.

Thanks a lot for mentioning that. We agree that there are different methods to mitigate catastrophic forgetting. However, in this work, our main focus is to show the performance of two widely used fine-tuning approaches in the source code OOD scenarios. We show that regardless of architectures and parameter sizes, the models' performance dropped in various OOD scenarios (Even for Code Llama 6B, which was fine-tuned using the LoRA method). The paper's main message is to raise awareness about the OOD issues of the fine-tuned LLMs in the source code domain.

In Section 4.4, what is the performance of the original pre-trained model without FT or LoRA to generate unseen language elements?

That is an interesting suggestion. We are working on the experiment to provide the results. We are trying to provide the results before the end of the discussion period.

[1] Nghi D. Q. Bui and Yijun Yu. Energy-bounded learning for robust models of code. CoRR, 2021.

Here, we provide the results for the following question. We hope that the description and the results resolve your concerns.

In Section 4.4, what is the performance of the original pre-trained model without FT or LoRA to generate unseen language elements?

In the following tables, we present the relative frequencies of generating unseen elements by models fine-tuned using both full fine-tuning (FT) and LoRA fine-tuning (LoRA) methods (The results from the paper - Figure 3). Furthermore, we included results of generating targeted elements (unseen elements for the fine-tuning cases) using the pre-trained models in zero-shot fashion to demonstrate their performance in generating target elements (Zero).

Table-1 and Table-2 present results for the text-to-code and code refinement tasks, respectively. Table-1 shows that for the zero-shot cases, the generated targeted elements by pre-trained models are on par with LoRA fine-tuning and have higher percentages than the full fine-tuning methods. In general, text-to-code tasks pose challenges, requiring models to map natural language text to targeted codes effectively. Consequently, pre-trained models may find it more difficult to generate desired outputs in this context.

In the zero-shot cases, Table-2 demonstrates that the pre-trained models generate more targeted elements than the fine-tuning methods. This shows that fine-tuning deteriorates previously acquired knowledge. Given the nature of the code refinement task, where the objective is to repair the input programs, models find it relatively easier to produce the programs (e.g., by copying the input tokens).

Note that, in all of the cases for text-to-code and code refinement tasks, the models' accuracy in the zero-shot cases was ~0.0% of relative performance (relative to the full fine-tuning results).

Table-1: The frequency ratios of the generated unseen language elements over the frequency in ground truth data for the text-to-code task. Zero refers to the results of the original pre-trained models. FT and LoRA refer to the results of full fine-tuning and LoRA fine-tuning methods, respectively.

Table-2: The frequency ratios of generated unseen language elements over the frequency in ground truth data for code refinement task. Zero refers to the results of the original pre-trained models. FT and LoRA refer to the results of full fine-tuning and LoRA fine-tuning methods, respectively.