The paper seems being written in a rush and has many syntax and expression errors. Line 36 and 37, economically and time-consuming; line 64, sperate -> separate; line 65 and 66, proprietary -> proper; line 93 boast -> boost; Figure 3, If the teacher model have better performance -> If the teacher model has better performance, the student model have -> the student model has; Line 67, Self-learning -> Self-Learning; Line 215, programmer , -> programmer; Line 265 Figure 5, Nature language -> Natural language; Line 351, We -> we; Line 378, large language models -> Large Language Models; Line 533, teaching stage -> Teaching stage, self-learning stage -> Self-Learning stage;

1. How to handle the case that the programmer cannot synthesize the correct code after multiple rounds of iteration? Purely discarding the unsuccessful responses will introduce much bias into dataset.
2. Following the above question, how to make sure the final synthesized coding questions and required codes possess enough diversity? Some coding questions may be too hard for GPT-4o to handle. Without human annotation, the synthesized dataset may have bias into simple and specialized questions.
3. The tested coding tasks are not hard enough. How about SWE-bench? Tasks like HumanEval have errors in some questions. Accuray difference of 2-5% may come from the dataset errors.
4. What is the intuition that the student model can perform better than the teacher model even just trained on the synthesized dataset without human annotation?
5. In the Self-Learning stage, the student model iterates by itself. Then how to make sure it can be better not even worse? For instance, some errors will be accumulated during dataset creation without human annotation. Some codes will be judged as correct but wrong. Some hard coding problems will never be generated since the model itself cannot iterate into this ability.
6. The results in Table 1 show nearly no improvements to the teacher model GPT-4. The proposed framework still seems as another kind of knowledge distillation from the larger model.
7. The theoretical analysis to prove better performance in Appendix Section B is not solid. For example, the assumption that iterative validation increases correctness is not such ideal. The error can appear in all the iteration steps, not decreasing over rounds.
8. Figure 3 and Figure 5 are not clear and easy for readers to understand.

1. Minor typos in the paper: the authors should use \citep{} for most of the citations.
2. I suggest the authors make more organized presentations in Figure 3 and Figure 4, and clearly state the process of how AIEV data is curated in the caption. This will make the method easier to follow and correlate with the text description in the main body.
3. Baselines comparison:
   • For Table 1, I wonder if the authors could provide results on tuning OSS-Instruct, i.e., Magicoder datasets?
   • For Qwen baselines, e.g., Qwen-2.5 in Table 1, I wonder if the authors could include their code-specific model, i.e., Qwen-2.5-Coder-Instruct, rather than the general model Qwen-2.5-Instruct?
   • The experiment section can be clearer if the authors provide some baseline setup at the beginning of Section 5 or state them more explicitly.
4. Clarification for Sec 5.5 and Figure 6:
   • The authors should explain what ST and MT datasets they are using. Are these datasets publicly available? Or is Figure 6 indeed an ablation study of the AIEV methodology (source data, MT dataset, and MT with execution)?

• Major Flaw: I don't see any direct, fair comparisons presented to previous data in a clear, transparent way.
  • L220: You seed data generation with Magicoder-Evol-Instruct and Magicoder-OSSInstruct.
  • To show that your data generation method is better than simply training on Magicoder-Evol-Instruct and Magicoder-OSSInstruct, you need to train a model on Magicoder-Evol-Instruct and Magicoder-OSSInstruct and compare it to the same model trained with the same settings on AutoCoder.
• L100: Claiming that "As far as we know, as of September 2024, AutoCoder is the only model that supports automatically installing external packages in the Code Interpreter" seems incorrect.
  • Any code interpreter that supports Jupyter notebooks (e.g. OpenCodeInterpreter or CodeAct) trivially supports arbitrary shell commands by emitting ! <any shell commands here>.
• I do not see any proof the self-training (a main claimed contribution) works.
  • To demonstrate that the self-training works, you would need to show some plot or table that shows accuracy before the self-training and accuracy after the self-training. By accuracy, I mean accuracy on some of the benchmarks you are testing.
  • For example, take a look at Figure 6 in this paper.

Summary

Each of the claimed contributions has problems, some of them severe. You have not showed that your data is better in a fair comparison, and there is no evidence (as far as I can tell) that the self-training works.

My review may seem very harsh, but I like the paper and the claimed contributions. I do not see evidence to support the claimed contributions. This surprised me, given that the empirical evaluation is otherwise extensive. Right now, the main contribution is that you have a very good OSS coding model — which is great — but you have not provided any evidence that the OSS model is great because of your pipeline and not because you trained a strong model on Magicoder-Evol-Instruct + Magicoder-OSSInstruct.

1. The writing and presentation are unclear and inefficient. Although the paper includes some details, most fail to offer a real understanding of the core algorithm, and some important information is still missing. Specifically, how is the data generated? Is it directly from the uncontaminated data you mentioned, with questions written based on the code? What prompts are used? Regarding the figures, most appear more as advertisements aiming to grab attention than thoughtfully designed visual aids. The fonts are oversized, and in Figure 3, lines 162-170 and 172-181 are exactly repeated.

2. I'm not sure what is the novelty based on the unclear presentation. Is this more efficient then previous methods? The writing seems to only focus on the engineering implementation.

3. Missing baselines. To demonstrate that Auto-Instruct outperforms previous methods like Evol-Instruct, experiments should include fine-tuning the same model, such as DeepseekCoder, on datasets generated by various methods with comparable size and generation cost, considering rewriting by GPT-4. However, no such results are provided.