OctoPack: Instruction Tuning Code Large Language Models

• One thing I find unclear about the writing is the name of CommitPackFT. As I understand, throughout the paper, the authors instruction-tuned the models using a 5000-sample subset of CommitPackFT, as mentioned on page 4

For instruction tuning our models, we select 5,000 random samples from COMMITPACKFT across the 6 programming languages that we evaluate on. Since you only used a subset of CommitPackFT to fine-tune the model, it might look a bit strange to call the entire dataset "FT".

1. This work finds out that OCTOCODER and OCTOGEEX, perform best among permissive models. I am curious about the reason behind that. Besides, a detailed introduction about OCTOCODER and OCTOGEEX would help understand the experimental results.
2. Still in experimental discussion, I am also curious about the performance of OCTOCODER and OCTOGEEX over the HUMANEVALEXPLAIN dataset. Why does OCTOGEEX have better performance in Go and Rust compared to OCTOCODER? What are the differences between OCTOCODER and OCTOGEEX?

The methodology makes a number of decisions that are not particularly well explained or defended. While none of these are critical, several of these would benefit from additional ablations and analyses, or an acknowledgement as a limitation. In no particular order:

The discussion around CommitPack leads with the rather enormous size of the initial dataset (~4TB). While this is accurate, the version used in this work is just 0.05% this size (2GB). This work offers no validation or experiments involving the original set. This both creates the impression that the 4TB set is likely to very noisy, and suggests that this work should really primarily emphasize the 2GB portion as far as concrete contributions of this paper are concerned. That would involve amending text like in the contributions on P2 and in the abstract, which don't mention the size of the subset that was actually used at all. This also relates to the "orders of magnitude more" comment on P9, which seems quite inconsequential.

Continuing the discussion of instruction data, it is not clear from the work why just 5K samples were chosen for fine-tuning. Was the concern that the model would drift too far if more samples were used? It is also not clear why (a) StarCoder was granted 3 extra samples, but that seems pretty inconsequential, and (b) OASST used nearly twice as many examples. Is that because the OASST samples are of higher quality? Was an ablation performed to choose these ratios? This is perhaps a particularly salient issue because one takeaway from this work is that CommitPackFT is not usable on its own. The experiments only ever report results of CommitPackFT + OASST (or similar combinations). This combination mostly boosts "Fix" performance; the results on the other two datasets are about even or worse than just using OASST by itself (Tab. 13). Perhaps the idea is that OASST is of very high quality but too small (are the ~8.5K samples used here the entire usable available subset?), so adding reasonably good samples increases performance? In any case, as is, the impression I get is that CommitPackFT isn't of particularly high-quality, but helps because it provides a bit more coding knowledge to OASST. More experiments could help contradict (or proof) that impression.

The work makes a rather strong argument, that building on HumanEval is a positive since it is so common that it is typically filtered from training data. That argument seems quite challenging to back up, given the absence of insight into how many LLMs are trained and anecdotal observations that pretrained models perform worse on new coding problems (e.g. from CodeContests). I would suggest a more moderated discussion of this choice, acknowledging the use of HumanEval as a potential limitation.

StarCoder is highlighted as performing very poorly in the explanation task, because it is unable to generate explanatory text. I wonder if this overlooks the obvious: query StarCoder to predict a docstring/javadoc/other comment using its FiM capabilities. The model has naturally not encountered samples were code is followed by a request for an explanation, but it would likely do relatively well when prompted to predict a comment above or at the start of a method. Although this somewhat stretches the definition of an "explanation", it does give the models more of a chance than the current setup.

Appendix C notes that inputs were filtered to just 768 tokens in length, for the complete before-code + message + after-code series. That is a very low limit by modern language modeling standards. Tab. 8 reinforces that this leads to very short programs compared to the natural distribution of commits. What was the motivation for these limits, and how does this impact the dataset's usefulness for downstream use-cases, where programs are rarely as short as those in the HumanEval suite?

Minor notes:

• The center panel in Fig. 3 might benefit from a visual improvement. Without reading the text, it was not clear to me what was happening here; as presented, it looks like the model is prompted with the code, predicts text, than is prompted with more text and predicts the code again, all in one input sample. Perhaps consider inserting a blank space between the two tasks (i.e., before the second model input) and moving the arrow to connect between the two halves.
• Sec. 3, P5: what experiments were conducted to confirm that GPT-4's accuracy only varies by 2% when sampling 1 sample vs. 20? I would imagine that the variation could be quite a bit more on some tasks. It might be worth sampling, say, 2-5 samples to balance cost and precision.
• It would probably help the work to also report BLEU/METEOR scores on the explanation generation task. This might offer a complementary view to the proposed automated metric, which is potentially somewhat lossy.
• A few odd notes from the appendix: why was "can’t you see i’m updating the time?" used as a prominent commit message filter (Tab. 5)? This doesn't seem to be a common phrase. Why was only data up till 2016 used? There are archives with more recent GitHub data.