Scaling Unlocks Broader Generation and Deeper Functional Understanding of Proteins

Thank you for your thoughtful feedback. We want to briefly correct the claim that “The authors train a suite of ProGen3 models ranging from 44M to 4.7B parameters” -- ProGen3 models range in size from 112M to 46B parameters. Addressing your points one at a time,

1. We have open sourced all model sizes up to 3B parameters, but we do not intend to open source PPA-1 or ProGen3-46B.
2. In Section 4, we do compare applying IRPO to ProGen3 against state-of-the-art supervised fitness prediction methods ProteinDPO [1] for stability prediction, and ConFit [2] and KERMUT [3] for general fitness prediction. However, we will add additional comparisons to ProGen2-XL [4], a 6.4B parameter dense autoregressive model, for both unconditional generation and alignment. We find that only only 4.1% of generations from ProGen2-XL (out of 4M total) are valid proteins (compared to 39.8% / 49.3% from ProGen3-3B/46B), and they cover 0.82x / 0.66x as many clusters as ProGen3-3B/46B. For a fixed parameter count, ProGen3 also outperforms ProGen2 on all our alignment benchmarks, but ProGen3 is much more efficient due to its sparse MoE implementation. We do not compare against ESM3 [5] because its license precludes us from using it, and we do not compare against AIDO.Protein [6] or ESM-IF1 [7] because they perform structure-conditioned generation, not unconditional generation.
3. Figures 1e and 4a only show a single point for 46B parameters because we only trained this model size on 1.5T tokens. Because we need to train a separate model for each dataset size (Lines 117-118), it was prohibitively expensive to train multiple versions of the 46B parameter model. Instead, we used our scaling laws to determine the right parameter/token split for 1.1e23 FLOPs.
4. We do not expect that masking random spans would yield an overly easy infilling objective. First, we still apply autoregressive attention to the full sequence, so infilling a span of length L is just as hard as generating the L residues at the C-terminal when performing N-to-C generation. Second, we train to infill long spans that can cover up to 80% of the protein and can be >400 residues (Lines 856-862). That said, your suggestions to apply fixed-region scaffolding or motif-conditioned design would be interesting extensions that could improve performance in fine-tuning regimes where more structural data is available; unfortunately, they are infeasible to apply on large-scale unlabeled datasets like PPA-1.

[1] Widatalla et al. “Aligning protein generative models with experimental fitness via Direct Preference Optimization.” bioRxiv, 2024.

[2] Zhao et al. “Contrastive Fitness Learning: Reprogramming Protein Language Models for Low-N Learning of Protein Fitness Landscape.” RECOMB, 2024.

[3] Groth et al. “Kermut: Composite kernel regression for protein variant effects.” NeurIPS, 2024.

[4] Nijkamp et al. “ProGen2: Exploring the boundaries of protein language models.” Cell Systems, 2023.

[5] Hayes et al. “Simulating 500 million years of evolution with a language model.” Science, 2025.

[6] Sun et al. “Mixture of Experts Enable Efficient and Effective Protein Understanding and Design.” bioRxiv, 2024.

[7] Hsu et al. "Learning inverse folding from millions of predicted structures." PMLR, 2022.

Thank you for your thoughtful feedback. Addressing your points one at a time,

1. In Section 4, we do compare applying IRPO to ProGen3 against state-of-the-art supervised fitness prediction methods ProteinDPO [1] for stability prediction, and ConFit [2] and KERMUT [3] for general fitness prediction. However, we will add additional comparisons to ProGen2-XL [4], a 6.4B parameter dense autoregressive model, for both unconditional generation and alignment. We find that only only 4.1% of generations from ProGen2-XL (out of 4M total) are valid proteins (compared to 39.8% / 49.3% from ProGen3-3B/46B), and they cover 0.82x / 0.66x as many clusters as ProGen3-3B/46B. For a fixed parameter count, ProGen3 also outperforms ProGen2 on all our alignment benchmarks, but ProGen3 is much more efficient due to its sparse MoE implementation. We do not compare against ESM3 [5] because its license precludes us from using it, and we do not compare against AIDO.Protein [6] or ESM-IF1 [7] because they perform structure-conditioned generation, not unconditional generation.
2. While it is true that the larger models in Section 3 were trained on more data, the point still stands that generation diversity improves when using more pre-training FLOPs. The mechanism is simple: a lower validation loss implies a better understanding of the natural protein distribution, especially for rare or out-of-distribution proteins (Lines 93-97). However, to address your concern more directly, we will create supplementary figures that compare the quality and diversity of generations from ProGen3-339M trained on 200B tokens (PG3-339M_200B), ProGen3-3B trained on 200B tokens (PG3-3B_200B), and ProGen3-3B trained on 500B tokens (PG3-3B_500B). We find that increasing ProGen3-3B’s dataset size from 200B to 500B tokens has a much smaller impact than scaling up from 339M to 3B parameters. 30.0%, 37.8%, and 39.8% of the generations are valid from PG3-339M_200B, PG3B-3B_200B, and PG3-3B_500B, respectively. Additionally, PG3-3B_500B covers 81% more clusters than PG3-339M_200B, but only 14% more clusters than PG3-3B_200B.
3. We are the first to systematically study the effect of the pre-training data distribution for PLMs, and we find that it has a significant effect on model performance (Figure 1d). Given that prior work has shown that the data distribution [8] and pre-training task [9] can affect scaling behavior, we feel that deriving compute-optimal scaling laws for this optimized distribution and for the infilling task are significant contributions. The fact that our results agree with Cheng et al. [9], who operate in a totally different setting, demonstrates the robustness of these scaling laws for autoregressive PLMs more broadly. We also performed the first wet lab experiment to characterize the impact of model scaling on the quality and diversity of generated sequences, and the findings in Section 3 are highly novel. Finally, while prior work has shown that preference optimization with wet lab data improves PLM performance, we are the first to show that larger pre-trained models yield better aligned models, especially in few-shot settings (Figure 4b).
4. IRPO can be used to increase the correlation of a model’s likelihood with any metric that we wish to optimize, but it is not suited for multi-class classification tasks. Section 4 shows that IRPO can greatly improve the correlation of ProGen3’s likelihoods with a wide range of protein properties. For general properties like thermostability, we show that aligned models can generalize to protein families that are highly distinct from those they were trained on, both for fitness prediction and sequence generation (Lines 251-254, Figures 4c-4g). We also show that for properties that are more specific to a protein family, including activity, binding affinity, and organismal fitness, as few as 500 experimental samples (a modest number that many labs can reasonably generate) can greatly improve the model’s performance for that specific task (Figure 4b, Supplementary Tables 8-9).
5. IRPO is a model-agnostic method. It is a variant of DPO that adds a negative log likelihood term to DPO’s contrastive loss (Appendix D.1). To strengthen our paper and address your concerns about having an apples-to-apples comparison with a different model family, we will include the results of applying IRPO to ProGen2-XL [4] in our revisions (see point 1 for more discussion of baselines). As mentioned above, we find that ProGen3 outperforms ProGen2 across the board. That said, similar to how IRPO is a variant of DPO that we found was well-optimized for ProGen3, ProteinDPO [1] is a variant of DPO optimized for stability prediction with ESM-IF1 [7], an inverse folding model that generates a protein sequence conditioned on its structure, and ConFit [2] is a variant of DPO that is optimized for ESM2 [10], a masked language model. Note that since masked language models don’t have well-defined likelihoods, vanilla DPO/IRPO cannot be applied to them. Moreover, we used identical train/val/test splits to evaluate each method. Thus, the baselines already in our paper are essentially the comparisons you are asking for, and we show that the aligned ProGen3-46B outperforms ProteinDPO and matches ConFit while being a much more flexible model that is capable of unconditional sequence generation.

[1] Widatalla et al. “Aligning protein generative models with experimental fitness via Direct Preference Optimization.” bioRxiv, 2024.

[2] Zhao et al. “Contrastive Fitness Learning: Reprogramming Protein Language Models for Low-N Learning of Protein Fitness Landscape.” RECOMB, 2024.

[3] Groth et al. “Kermut: Composite kernel regression for protein variant effects.” NeurIPS, 2024.

[4] Nijkamp et al. “ProGen2: Exploring the boundaries of protein language models.” Cell Systems, 2023.

[5] Hayes et al. “Simulating 500 million years of evolution with a language model.” Science, 2025.

[6] Sun et al. “Mixture of Experts Enable Efficient and Effective Protein Understanding and Design.” bioRxiv, 2024.

[7] Hsu et al. "Learning inverse folding from millions of predicted structures." PMLR, 2022.

[8] Mayilvahanan et al. “LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws.” ICML, 2025.

[9] Cheng et al. “Training Compute-Optimal Protein Language Models.” NeurIPS, 2024.

[10] Lin et al. “Evolutionary-scale prediction of atomic-level protein structure with a language model.” Science, 2023.

Thank you for your thoughtful feedback and the highly relevant references. We did some early experiments with offline preference learning algorithms like LiPO [1] that make full use of the continuous fitness values, but we found that they generally underperformed DPO/IRPO. Prior work has also shown that contrastive ranking losses can be more effective than regression losses for protein variant effect prediction, especially in few-shot settings like those we consider in Figure 4b [2-3]. However, we agree that moving beyond ranked pairs is an important direction for future research in aligning protein language models with continuous-valued experimental data.

Your points about the gap between predictive & generative capabilities and between online & offline algorithms are interesting ones. We do find that aligning on the Megascale stability dataset allows us to generate proteins that are more stable both computationally (Figure 4f) and experimentally (Figure 4g); additionally, larger models generate more stable proteins than smaller models (Figure 4f), matching the trends of the stability prediction results (Figure 4c-e). A key difference between our setting and the NLP one is that we select <100 low-perplexity sequences out of 50,000+ generated candidates, while in NLP, LLMs rarely generate >100 candidates. Our setting may therefore increase the importance of the model’s ranking capabilities and mitigate some of its generative shortcomings.

We chose not to use an online algorithm due to the additional complexities it introduces: we would need to train a different reward model for each dataset, ensure that reward model generalized from the local mutational landscapes it was trained on to model generations, and fine-tune and/or prompt ProGen3 for each dataset to ensure that we sample generations from the right protein cluster. Given that we trained 4 models on 9 datasets each, it was impractical to fully tune all these hyperparameters for each setting, but comparing online and offline methods (especially in settings where the generations from each can be tested in the lab) is an exciting future direction.

We are more than happy to address your minor points, especially if you have any specific feedback on what would make Figure 1a more interpretable.

[1] Liu et al. “LiPO: Listwise Preference Optimization through Learning-to-Rank.” NAACL, 2025.

[2] Brookes et al. “Contrastive losses as generalized models of global epistasis.” NeurIPS, 2024.

[3] Zhao et al. “Contrastive Fitness Learning: Reprogramming Protein Language Models for Low-N Learning of Protein Fitness Landscape.” RECOMB, 2024.

Thank you for your thoughtful feedback. Addressing your points one at a time,

1. As you mentioned, we have open sourced all model sizes up to 3B parameters, but we do not intend to open source PPA-1 or ProGen3-46B.
2. As models grow larger, they improve at estimating the distribution of naturally occurring proteins. However, since the forces driving evolution don’t always correlate with the experimental read-outs of laboratory assays, the average protein found in nature is rarely the most optimal. Thus, effectively modeling the evolutionary distribution of proteins can be at odds with zero-shot fitness prediction, since the most optimal variants may be quite unlikely in nature. Others have also theorized about and documented this trend [1-2], and we allude to their work in Lines 219-226. Rather, we find that larger models learn better representations that are more useful for downstream tasks.
3. Based on our findings, we expect that it is actually quite easy for ProGen3 models to generate expressing (but potentially non-functional) proteins, provided that those proteins come from families that are in-distribution for the model. Consequently, unconditional generations from all models have similar expression rates when compared head-to-head, but larger models trained on more data can generate expressing proteins for families that are out-of-distribution for the smaller models. For the infilling task, we pre-selected protein scaffolds for the model to infill, and a number of these scaffolds happened to be in-distribution for ProGen3-46B but out-of-distribution for ProGen3-3B (but not vice versa).

[1] Weinstein et al. "Non-identifiability and the blessings of misspecification in models of molecular fitness.” NeurIPS, 2022.

[2] Gordon et al. “Protein language model fitness is a matter of preference.” ICLR, 2025.