FILI: Syntax Repair By Learning From Own Mistakes

The paper proposes Fili, an to training syntax repair neural networks. Fili works by iteratively attempting to repair a set of broken programs, then adding all pairs of (original broken program, fixed proposed program) and (still broken proposed program, fixed proposed program) to its dataset, re-training on the new dataset, and repeating. The paper augments this by proposing a curriculum learning approach, in which the neural network is presented with (broken, fixed) program pairs with larger and larger edit distances over the course of training. The paper evaluates Fili against the prior state-of-the-art baseline, Bifi, and finds that Fili improved on Bifi by about 1%. The paper also evaluates Fili against LLMs prompted to perform syntax repair, finding that GPT-3.5 generates parseable programs more often than Bifi, but with a significantly higher edit distance (making other program changes).

This work proposes a FILI (fix-it-learn-it) method to train the model for fixing syntax errors in programs. It improves over the existing BIFI approach, without having to train any additional models for data augmentation. As a result, in each iteration, FILI finetunes the fixer model by its own prediction.

The paper presents a new technique for repairing syntax errors. The goal is to train a fixer model, one that takes a "bad program" that does not parse, and outputs a "good program" that parses correctly.

To control for arbitrary modifications, evaluation only considers edits that are up to some fixed edit distance from the original "bad" program.

Previous work (BIFI) used a combination of a Fixer/Breaker models to train the Fixer model (and simultaneously, a Breaker model used to generate incorrect examples that are similar to real-world bad programs).

In contrast, this work (FILI) only uses a single Fixer model, and uses negative samples from the Fixer's beams to augment the data used to fine-tune the fixer. FILI uses high-confidence incorrect predictions from the highly-ranked beams as negative examples to be paired with the correct program (one that parses). This is similar to the approach taken by (Cao et al. 2021).

The new approach shows a modest improvement over BIFI, but does that when only using a single Fixer model.

This paper proposes FILI (Fix-It-Learn-It), which simplifies a previous work -- BIFI (Break-It-Fix-It), an unsupervised learning approach for fixing syntax errors in programs. BIFI requires two trained models (i.e., a breaker and a fixer), while FILI requires only one fixer model. The observation is that the fixer model is not perfect and thus generates correct fixes as well as incorrect fixes. In the latter case, the fixer model itself can be viewed as a breaker model. Instead of training a separate breaker model, which can be expensive, one fixer model can be used to generate both good programs and bad programs. The evaluation on the same dataset shows that FILI slightly outperforms BIFI.

We thank the reviewers for their helpful comments and suggestions. We will incorporate all suggestions and clarify the confusion in our next version. Below, we address the concerns that the reviewers raised.

Should be "FILI cannot solve 1263, while BIFI cannot solve 1428"?

Thank you for pointing this out, it was a small typo and indeed it should be FILI cannot solve 1263, while BIFI cannot solve 1428.

LLMs experiment: instruction for making only local changes

We prompt the model to make only local changes to the program and not make any other changes. We use the following two prompts and evaluate both Bard and GPT-3.5 on the test-set:

Prompt-1:

'''

The following python program has syntax errors. Fix all the syntax errors to make the program parsable. Do not make any other changes to the program. Just fix the syntax errors. <Incorrect Program>:

{{incorrect}}

<Correct Program>:

'''

Prompt-2:

'''

The following python program has syntax errors. Fix all the syntax errors to make the program parsable. Do not make any other changes to the program. Keep the edit distance between the original program and the fixed program as small as possible. <Incorrect Program>:

{{incorrect}}

<Correct Program>:

'''

Despite instructing the models not to modify other sections of the program through prompts, the edit distance between the predicted program and the incorrect source program exceeds the threshold used by BIFI (and FILI) for evaluation. Consequently, although LLMs are good in fixing syntax errors, they introduce changes to other parts of the program.

Accuracy gains are small

We agree with the reviewers that FILI does not significantly outperform BIFI in terms of accuracy. With FILI our major contribution is significantly simplifying the training of fixer in an unsupervised setting while still achieving slightly better accuracies compared to BIFI, the previous state-of-the-art approach for training syntax repair models. For training FILI, it took ~20 hours to train with one TPU (v3-8) and for training the full BIFI model with the breaker training it takes ~72 hours. BIFI involves training the fixer and the breaker model which is expensive. Moreover, the BIFI framework also involves making an inference on ~3 million synthetic data examples using the trained breaker model to augment data for the fixer training which can be very expensive. In contrast, FILI uses the beam predictions to augment data which is significantly cheaper to perform than doing an inference on the synthetic data. Overall FILI’s approach is more resource efficient while maintaining the similar accuracies as the previous models. The efficiency in training is particularly important in light of the fact that obtaining compute is challenging.

Cost/difficulty of training breaker model in BIFI:

BIFI trains the breaker model with significantly more data than the fixer model. The training of the breaker model involves using the <good programs, bad programs> pair. To generate the training data, BIFI applies the breaker model to the good programs, producing the corresponding incorrect programs. The overall training of the breaker model takes more time compared to the fixer because there are 3 million good programs (syntactically correct) on which the model needs to perform inference to generate additional data. This inference must be carried out for each training iteration of BIFI, as the breaker model is updated after every iteration, resulting in an expensive training procedure. On the other hand, the fixer's additional training data is generated by performing inference on only ~20k examples, leading to a more cost-effective training procedure than the breaker model.

The authors unanimously raised a couple of concerns, including (1) relatively incremental improvements in some cases, and (2) a relatively reasonable question about whether the tasks considered in the paper are solvable by well prompted LLMs. While the authors point that LLMs take significant additional resources to train is well taken, ultimately these models do exist. The authors point out that the LLMs they tested made code edits in unintended areas, which would be an interesting limitation. However, I'd like to see this limitation studied more systematically with experimental setup given. The authors mention that the instructed the LLMs to not do this, but it's not clear if this problem couldn't be solved, for example, by simply constructing a better prompt. Ultimately, I think it's fair to want these questions answered, and the paper would be much stronger for it.

Reject