Sidechain conditioning and modeling for full-atom protein sequence design with FAMPNN

We thank the reviewer for their insightful comments and questions. Below, we address these questions in detail:

FAMPNN sampling time comparisons

Please refer to the last section of the response to reviewer GPV6

Standard deviation in the predictions

Please refer to the first section of the response to reviewer MU8A

Supervised baselines for Figure 5

This is an excellent suggestion, for 5c there are baselines available with formal test splits for stability datasets on which we can evaluate FAMPNN to compare to supervised methods. Supervised baselines and formal test splits do not exist for 5a, but we can run our own baseline (eg. one-hot-encoding of sequence with a ML-based regressors). If accepted, we will include these additional results.

ProteinMPNN self-consistency on longer lengths

This is an interesting hypothesis that we also explored. We believe that side-chain prediction error may indeed be compounding for longer-length proteins. At each sampling step, we condition only on previously-predicted side chains with predicted error < 0.3Å, and notice a larger performance degradation without this filtering.

Why explictily predict sequence?

We agree that given a full-atom representation, it is trivial to predict the sequence as it is already defined by the atom composition. In early versions of the model, we experimented with removing explicit sequence prediction and calling sequence from the implicit full-atom predictions, but we found this to not work as well in practice. We believe that the cross-entropy loss objective is important for the model to learn to predict high-quality sequences. Additionally, in addition to careful tuning of our masking schedule, and occasionally completely dropping all side-chain coordinates, FAMPNN is able to predict sequences with no full-atom representation, allowing use in situations such as sequence design for de novo backbones.

Multi-chain sequence design with FAMPNN

Please refer to the rebuttal to reviewer GPV6 for details on the training of the PDB model and the respective dataset. Regarding multi-chain design, we showcase the capability of the PDB model for multi-chain design in Figure 6b, where we demonstrate the utility of full-atom context for higher sequence recovery in protein-protein interfaces. Here, FAMPNN achieves higher sequence recovery, and higher sequence recovery scaling with context than LigandMPNN. Additionally, we note FAMPNN performs state-of-the-art in binding-affinity fitness evaluation in large multi-chain protein-protein complexes greater 3000 residues (SKEMPIv2 dataset), which are shown in figures 5a and 5c. Together these results demonstrate robust utility of FAMPNN in multi-chain design.

We'd like to thank the review for their positive evaluation and interest in the methodological choices behind our model. Below, we clarify these points:

Ghost atoms for masked tokens

To encode the ghost atom for masked tokens, we set them at the position of the central CA atom by default. Because the model receives a mask sequence token for this position, it is able to understand that these ghost atoms refer to a masked position.

MLM vs. AR

In early versions of the model, we experimented with both AR and MLM, finding that they perform similarly in terms of both sequence recovery and self-consistency. We noticed that high-quality sequences could be predicted in relatively few steps, so viewing the generative process through an MLM framework could allow for faster inference. We also chose MLM because it gives us more flexibility in choosing train-time masking schedules, allowing the model to prioritize learning on certain masking levels.

We appreciate the reviewer’s comprehensive evaluation and suggestions to improve clarity and benchmark comparisons in our paper. Below, we provide detailed responses and clarifications:

Inclusion of perplexity in addition to sequence recovery for evaluation

This is a great suggestion. With 1 step recovery, FAMPNN achieves a perplexity of 4.99, and we will update this with the perplexities of all methods in Table 1 in a revised version.

Evaluation on ProteinGym

We agree with the reviewers point and will include ProteinGym in our evaluations if accepted. We would also like to note that many of the assays included in our fitness evaluations make up a great amount (but of course not all) of ProteinGym data (eg. Megascale, SKEMPIv2).

Interval estimates

We thank the reviewer for pointing this out and believe it would be a valuable addition. We intend to add interval estimates for performance metrics in the final version if accepted.

Expected monotonic relationship between context and packing accuracy

We believe that the slight deviations from this trend are due to random chance, and we plan to update this plot with error bars as well as clarify this claim in the text.

Selection of benchmark models

In general, we chose to benchmark models that are commonly used for their respective tasks. We agree that ESM3 should be included in the zero-shot fitness prediction evaluations, and we will include this in the final version if accepted.

Novelty of our work in the context of Chroma

As correctly pointed out, Chroma does in fact have the capability to return an all-atom structure given an input backbone via its ChromaDesign module. However, the sequence design module does not have the ability to encode a full-atom protein structure and therefore cannot design sequences conditioned on neighboring sidechain context. As a consequence, this means that Chroma's sequence predictions are unable to leverage either experimentally determined sidechain conformations or previously predicted sidechains. By contrast, FAMPNN explicitly conditions on sidechain context, which allows users to provide known sidechain conformations as conditioning input during sequence generation. This allows for more fine-grained control. Despite this, we very much agree it would be good to discuss Chroma and benchmark ChromaDesign along with other methods. We will provide updates in a revised version and have included an extension to Table 5 below:

We also provide an extension to Table 2 for sidechain packing evaluation below, noting that Chroma has not been trained on a dataset to explicitly hold out CASP homologues:

Potential mismatch for antibody-antigen binding dataset evaluation

Yes this is true. As described in Shanker et. al, experimental structures matching the exact sequence are not available for each assay, however as the sequences of the different HA variants (H1,3,9 etc.) are quite close, models are still able to perform well using the backbone of the homologous proteins. While this can bring about errors, and we do acknowledge this, this is an extremely common occurrence in practical protein design, where homologous structures must be used due to a lack of experimental structure (homology modelling). Thus, we see this as an opportunity to showcase the model's performance in a structurally data limited situation which is even more prudent to demonstrate given our method uses “more” structure than its backbone-only counterparts.

Other comments or suggestions

We thank the reviewer for suggestions on improving the clarity of our figures and will make these changes if accepted.

We thank the reviewer for their suggestions to strengthen our baselines and evaluations. Below, we address the raised concerns:

Inclusion of more recent sequence recovery baselines

We thank the reviewer for pointing out this omission. To this end, we report Frame2Seq (Dec. 2023), a structure-conditioned masked protein language model achieve 46.53% sequence recovery as a single model, and 49.11% as an ensemble of three models on the CATH 4.2 test set with a single step (Akpinaroglu et. al), while FAMPNN achieves a 49.66% and 50% recovery as a single model with single and 5-step sampling respectively. Second, some other baselines could not be included due to differences in training data: e.g. certain methods do not train on CATH and elect to train on non-standardized versions of the PDB, thus we cannot compare sequence recovery on a common test set. Third, certain recent baselines for sequence recovery, particularly hybrid models which utilize pre-trained protein language models such as ESM2 (e.g. LM-Design, KW-Design), may have data-leakage issues with the CATH 4.2 test set, as pretrained protein language models likely have trained on sequences in the CATH test set. We will include a discussion of this in the final version of the paper if accepted. Finally, in this paper, we note that we do not claim state-of-the-art performance in terms of sequence recovery, but rather that sidechain packing and sequence design are synergistic tasks to learn, where we can achieve self-consistency competitive with ProteinMPNN while achieving state-of-the-art packing.

High sequence recovery ≠ good design

We completely agree that high sequence recovery does not equal good design. To this end we use the de novo benchmark, popularized by ProteinBench (Ye et. al), as a way of measuring the ability to recapitulate structures on structures that are structurally distinct and diverse from those found in nature and report self-consistency to capture if designs “fold into the target structure” as you mention. As these proteins are not found in nature, we hope this demonstrates generalization beyond memorization. In fact, we find very low average pairwise similarity between sequences generated by FAMPNN on this benchmark with [13.4%, 9.1%, 8.4%, 7.9%, 7.8%] average pairwise similarity for de novo backbones of length 100, 200, 300, 400, and 500 respectively. We will conduct a more robust benchmark of diversity on de novo backbones and compare to other methods in the final version of the paper if accepted.

Training and inference costs

We completely agree that compute costs are important for a useful protein design tool. Our CATH and PDB models are competitive with state-of-the-art models at 8.6M and 10.7M parameters respectively, within magnitude of models such as ProteinMPNN (1.9M), Frame2Seq (10M), and PiFold (6.6M). Notably, our model is much smaller compared to models such as ESM-IF (142M), LM-Design (664M), and ESM-3 (1.4B). For training, we mention in Section B.2 that we trained our CATH model for ~8 hours on a single H100 GPU with 80GB of memory. The PDB model was trained with 4 H100s for 72 hours, but in practice we notice that the model can reach similar performance in ~24 hours (96 GPU hours). Regarding example size, training of our PDB dataset was conducted with examples cropped or padded to a large fixed size of 1024 residues, with 82.91% of examples being multi-chain complexes. Finally, regarding inference costs, when evaluating our CATH model on the test-set, we achieve a single-step sampling speed of 0.03s and 0.11s per sample on a single H100 GPU. Additionally, we are enthusiastic to add a detailed comparison of inference costs to other methods to the final version of the paper. We appreciate this point being, as we have found that because we use a MLM procedure, FAMPNN can sample equally high quality sequences with much fewer steps, making our method much faster for longer proteins than other methods, including ProteinMPNN.