SpecMER: Fast Protein Generation with K-mer Guided Speculative Decoding

We thank you for your feedback. Your constructive comments have helped us clarify key points and strengthen our paper.

Limited evaluation. The method is evaluated on five proteins, three of which are shorter than 100 amino acids. I think that a more diverse selection of proteins should be examined. I also think that the inclusion of a diversity metric (such as sequence identity or similarity to the wild type) or the inclusion of additional sequence quality metrics would further demonstrate the improved generation.

We have performed additional experiments on two longer proteins: CBS_HUMAN (551 amino acids) and ADRB2_HUMAN (413 amino acids). We report the average, top-20 and top-5 NLL:

ADRB2_HUMAN

CBS_HUMAN

We found that SpecMER outperformed speculative decoding, further supporting our existing results. We will include these results in the main text.

We agree that diversity metrics help strengthen our findings. For each protein, we computed the average Hamming distance between generated sequences and the wild-type sequence. Additionally, we computed the average pairwise Hamming distance between generated sequences to measure inter-sequence diversity:

SpecMER generates sequences far away from the wild-type while maintaining inter-sequence diversity, exploring sequence space while still generating plausible protein sequences.

Limited comparison. The quality of the generated sequences are only compared between different hyperparameter configurations of SpecMER. Since the intention is to replace standard autoregressive decoding, I think that the generation quality should be directly compared to the sequences generated by the draft and target models, potentially at different temperature values.

SpecMER, being a type of speculative decoding, increases the generation speed with no sacrifice to the quality of the sequences (Leviathan et al. (2021), Sun et al. (2022)). We discuss this property and expand on its result in section 2 and 3, but are happy to discuss in more detail in the camera ready version.

To demonstrate this property, we conducted additional experiments generating sequences using only the target model (ProGen2-M) for 4 hours to compare sequence likelihoods to SpecMER (c=5) using the same temperatures as the results in Table 2. The resulting sequences were scored using ProGen2-M, and we report the top 20 highest likelihood averages:

We found that protein sequences generated by SpecMER (c=5) had likelihoods comparable to, or exceeding, those from the target model, a result we discuss in Section 3.3.

Limited impact. Without the above analyses, the model presumably yields similar quality sequences ~30% faster. While this is certainly useful, the comprehensive hyperparameter sweep generates a total of 7200 sequences which is not insubstantial. I think that the benefits of using SpecMER compared to simply sampling from the target model at different temperatures for a comparable amount of time should be clearly demonstrated in order to convincingly show the impact.

We perform an extensive hyperparameter sweep, which we detail in Appendix D, to identify the best configuration per protein, a one time cost that results in accelerated generation. As noted in our previous response, we performed additional experiments comparing SpecMER (c=5) to standard autoregressive decoding, using the same temperatures of the results in Table 2. SpecMER produced sequences with likelihoods comparable to or exceeding standard autoregressive generation, supporting that generation quality is maintained while achieving significant speedups.

L146-147: As per L140, the k-mer scoring function scores c sequences of length L. In that case, shouldn’t the list be over the length L sequences, i.e., …,

You are correct. The way it is described in the paper could be clarified. c x L indicates the batch dimension, i.e. c sequences each of length L.

Section 4: It is not immediately clear to me what the final hyperparameter configuration actually is. In section 4.2 and in the appendix, it is stated that all combinations of hyperparameters are evaluated.

The final hyperparameter configurations for each protein reported in Table 2 are:

We will include these values in the Appendix.

Does this mean that all 36 configurations are used to generate 200 sequences each, and the single configuration with the best results are then chosen and presented in the tables?

Yes.

Is the termination criteria of stopping when the wild-type sequence length has been reached sensible? Or could this have negative effects on sequence (and the resulting structure) quality?

Yes. Our aim is to design variants around the wild-type sequence, a common practice in protein engineering (Hayes et al., 2025). In practice, we observed no degradation in sequence quality - sequences matched or exceeded the quality of those generated with standard autoregressive decoding.

What would the metrics look like if the exact wild-type sequence was sampled?

Here is the negative log-likelihood and pLDDT for each wild-type sequence:

What is the complexity of SpecMER w.r.t. sequence length and hyperparameters?

SpecMER has the same computational complexity as speculative decoding, O(L^2), with respect to sequence length L. It does not introduce additional overhead beyond speculative decoding, whose accelerated generation is achieved without increasing asymptotic complexity relative to standard autoregressive decoding (Leviathan et al., 2021). The additional k-mer scoring in SpecMER is constant time and does not affect overall complexity. We will make this clear in the main text.

How does MSA depth affect the quality of the generated sequences? Since only five proteins (three of which are very short) are examined, I’m not yet convinced that MSA depth and quality is independent.

We ran additional experiments to test the importance of MSA depth for Bgl3, forming k-mers using 1,000 sequences from the MSA instead of the full depth MSA (130k sequences), limiting the guidance signal from the MSA. We observed a sharp decline in likelihoods, with the top-20 sequences yielding a likelihood of 1.56 +/- 0.20, whereas SpecMER (c=5) with the full depth MSA yielded an average of 0.63 +/- 0.11. This supports our claim in the conclusion that performance may degrade when informative motifs are sparse or unavailable. We will include an updated section in the appendix with this ablation.

Which ESM-2 version was used to compute the embeddings for the PCA? Were the embeddings mean-pooled? Are the aligned sequences embedded, or are they converted into unaligned sequences? I think some additional details should be included in appendix C.1.

We utilized ESM-2 8M with mean-pooled embeddings and unaligned sequences in accordance with standard practice. We will update Appendix C.1 to reflect this.

How does context length affect generation quality? The context length is chosen to be approximately 10 % of the wild-type sequence length. Why is that?

We chose a context length of ~10% based on empirical observation: shorter contexts often led to pathological repetition of amino acids (Hsu et al., 2022), while longer contexts restricted exploration of sequence space. A 10% context provided a balance, enabling high-quality, diverse sequence generation.

In the introduction, ProGen2-XL is used as an example, yet the chosen target model is ProGen2-M. What would the speed up using SpecMER be if ProGen2-XL was used instead?

We ran additional experiments with ProGen2-XL as the target model and ProGen2-M as the draft. The tokens per second jump from 7.03 to 9.72 with SpecMER (c=3), a 38% increase in speed.

We will address your formatting concerns while making revisions. Thank you again for your feedback.

Thank you for your constructive feedback. We have touched on some key points of clarification to address your comments.

It’s unclear what the main contribution of the paper is. From the abstract, it feels like SpecMER is the main contribution, which is built on top of existing speculative decoding methods. However, while reading the paper, if I understood correctly, speculative decoding was not used at all for protein generation. If this is the case, I think it should be emphasized more clearly in the abstract.

To clarify our contributions:

We adapt speculative decoding to protein language models and demonstrate acceleration over standard autoregressive decoding (Table 4, Baseline). Prior works in speculative decoding have focused on natural language. To the best of our knowledge, speculative decoding has not previously been applied to protein generation, so this is a key contribution of our work.

We introduce SpecMER, maintaining speedups from speculative decoding, while generating protein sequences with higher biological plausibility. We discuss these contributions in lines 60-70, but we are happy to make it more clear in the abstract.

The following drawback follows from my previous comment: there are two metrics to evaluate model performance—speed and accuracy. However, there is a kind of inconsistency in the comparison. To show speed-up, you compare SpecMER with the target model and demonstrate improved performance, which is fair. However, in terms of accuracy, you compare only speculative decoding against SpecMER. If both of these methods are introduced by you, I don’t think this proves SpecMER’s efficiency convincingly. A comparison should also be made with existing methods—for instance, the one you used as a baseline for demonstrating the speed-up.

SpecMER, being a type of speculative decoding, increases the generation speed with no sacrifice to the quality of the sequences (Leviathan et al. (2021), Sun et al. (2022)). We discuss this property and expand on its result in section 2 and 3, but are happy to discuss in more detail in the camera ready version. To demonstrate this property, we conducted additional experiments, generating sequences using only the target model (ProGen2-M) for 4 hours to compare sequence likelihoods to SpecMER (c=5). The resulting sequences were scored using ProGen2-M, and we report the top 20 highest likelihood averages:

We found that protein sequences generated by SpecMER (c=5) had likelihoods comparable to, or exceeding, those from the target model, a result we discuss in Section 3.3.

Could you add more specific details relevant to the fact that this work is being submitted to a computer science conference? It currently feels like it lacks a clear description of the model architecture, training protocol, and computational resources used—I'd love to see that information, perhaps included in the appendix.

ProGen2 is a decoder-only Transformer architecture with four different parameter sizes: small - 151M; medium/base - 764M; large - 2.7B; xlarge - 6.4B. No model training is performed in this work; inference is run on an NVIDIA A6000 GPU. More details can be found in Nijkamp et al. (2022). We discuss some of these details in Section 4, but are happy to add more details in the appendix.

Also, looking at Table 4, it seems that increasing the c parameter in SpecMER leads to a noticeable increase in variance. Could you comment on why you think that might be the case? And what might be the concerns of using high c?

This increase in variance is due to batch generation with ProGen2. Batch generation is slightly slower than producing a single sequence continuation, and the added time to generate increases with c. Therefore, when a speculated token is rejected, decoding takes longer and the added cost of batch generation appears in the variance. Due to the non-deterministic nature of sampling, some generations have higher acceptance rates than others; therefore, occasional rises and dips in acceptance have a larger impact on tokens per second than producing a single sequence at a time. Further discussion on this trade-off space can be found in Appendix B.2.

Thank you for your constructive feedback. After reviewing your comments, we have touched on some key points of clarification.

Because computational sequence generation already outpaces wet-lab synthesis and screening by several orders of magnitude, the claim that “slow sequence generation can delay high-throughput design workflows by days” appears overstated and insufficiently justified.

The quality of designs chosen for wet-lab synthesis can drastically speed up the design process by proposing better variants, shortening or skipping design cycles altogether (Biswas et al., 2021). SpecMER addresses this by generating biologically plausible sequences with higher likelihoods and pLDDT scores than standard autoregressive decoding (Section 4.3). By producing more high-quality sequences in the same compute time, SpecMER increases the likelihood of finding stronger candidates earlier, ultimately reducing the experimental effort required during wet-lab synthesis.

Improvements in negative log-likelihood and pLDDT are modest, and their biological significance is not demonstrated.

The goal of this work is to introduce a faster decoding method that can be applied to existing PLMs. We demonstrate that we can increase decoding speedups without degrading output quality, and in many instances, produce a better output. We believe this is a valuable contribution to the design process, particularly in scenarios where speed is critical (i.e. producing large libraries of candidate sequences). We note that even modest increases in negative log-likelihood and pLDDT scores will result in improved structural quality (Table 3) and can result in drastic improvement in function. Our results demonstrate that SpecMER achieves these improvements while enabling faster protein generation and accelerating protein design workflows.

The study lacks ablations with random k-mer tables or shuffled MSAs (or MSA from another protein), making it hard to confirm that observed gains come from genuine evolutionary signal rather than extra sampling.

We performed two ablation studies to test the efficacy of k-mers. We first tested SpecMER using two configurations: generate conditioned on GFP, use GB1-derived k-mers to select continuations; generate conditioned on GB1, use Bgl3-derived k-mers to select continuations.

This mismatch in evolutionary signal led to lower likelihoods both on average and for top end generation (top-20) when compared to Table 2, indicating that protein specific k-mers are responsible for the increase in likelihood. The second ablation tested the importance of MSA depth. We generated Bgl3 proteins using 1,000 sequences from the MSA instead of the full depth MSA (130k sequences), limiting the number of k-mers to guide generation. We observed a sharp decline in likelihoods, with the top-20 sequences yielding a likelihood of 1.56 +/- 0.20, whereas SpecMER (c=5) with the full depth MSA yielded an average of 0.63 +/- 0.11. This further supports our claim in the conclusion that performance may degrade when informative motifs are sparse or unavailable. We will include an updated section in the appendix with this ablation.

Novelty is a key criterion in protein generation. Does incorporating MSA-derived information make the generated proteins more similar to existing sequences?

To evaluate novelty, we computed the Hamming distance between each generated sequence and the wild-type, as well as the pairwise Hamming distance among generated sequences (see table below).

SpecMER generates sequences far from the wild-type while maintaining inter-sequence diversity, exploring novel sequence space. Additionally, Appendix D illustrates embedding plots comparing generated sequence to MSA sequences, indicating that SpecMER generates novel, high-likelihood sequences that are distinct from the wild-type yet remain grounded near homologous sequence space. We will update the appendix to include these results.

Thank you for your thoughtful and detailed feedback. Your constructive comments have helped us clarify key points and strengthen the paper.

Authors should include a few additional citations to increase the quality of their scholarship and expand upon why generation from autoregressive models is challenging.

We will include the following references: (Hsu et al., 2022), (Spinner et al., 2023), (Holtzman et al., 2020). Additionally, we will expand upon challenges in autoregressive decoding in the introduction, including pathological repetition, early termination, and OOD generation.

Have the authors checked if key residues in the proteins they design are preserved or accurately sampled?

Our work does not explicitly evaluate the preservation of individual residues, as the core objective of SpecMER is to accelerate protein generation while maintaining or improving overall sequence quality. Our results demonstrate that proteins generated by SpecMER achieve higher likelihoods and pLDDT scores than those of speculative decoding, suggesting that the generated sequences align with commonly used design proxies.

What is the tradeoff between 13K sequences at original speed vs 20K sequences at faster speeds?

SpecMER, being a type of speculative decoding, increases the generation speed with no sacrifice to the quality of the sequences; therefore, there is no inherent tradeoff (Leviathan et al. (2021), Sun et al. (2022)). We discuss this property and expand on its result in section 2 and 3, but are happy to discuss in more detail in the camera ready version. We conducted additional experiments generating sequences using only the target model (ProGen2-M) to compare sequence likelihoods to SpecMER (c=5). The resulting sequences were scored using ProGen2-M, and we report the top 20 highest likelihood averages:

How does ProGen-XL, -S, -M compare in speed to each other and how does that compare to other autoregressive models that exist? Having a 32% speedup is great, but if other AR models are already faster then wouldn't the protein designer just want to use those? No cross-model speed benchmarks are included, and no papers are cited that provide such benchmarks either.

We tested the generation speed of ProGen2-S, M, and XL. ProGen2-S: 74.11 toks/s, ProGen2-M: 31.48 toks/s, ProGen2-XL: 9.72 toks/s. We observed a sharp decline in speed, reinforcing the difficulties of generating large amounts of sequence with autoregressive generation. We then performed experiments with Tranception and found that it generated sequences more slowly than ProGen2 and with lower likelihoods. As SpecMER is a decoding strategy for accelerating autoregressive generation, it is applicable to any AR model given a compatible vocabulary size.

Can you provide more experiments to back up the claim "The effectiveness of SpecMER depends on the quality of input alignment (MSA); performance may degrade when informative motifs are sparse or unavailable"?

We performed two ablation studies to test the efficacy of k-mers. We first tested SpecMER using two configurations: generate GFP, use GB1-derived k-mers to select continuations; generate GB1, use Bgl3-derived k-mers to select continuations.

This mismatch in evolutionary signal led to lower likelihoods both on average and for top end generation (top-20) when compared to Table 2, indicating that protein specific k-mers are responsible for the increase in likelihood. The second ablation tested the importance of MSA depth. We generated Bgl3 proteins using 1,000 sequences from the MSA instead of the full depth MSA (130k sequences), limiting the number of k-mers to guide generation. We observed a sharp decline in likelihoods, with the top-20 sequences having a likelihood of 1.56 +/- 0.20, whereas SpecMER (c=5) with the full depth MSA had an average of 0.63 +/- 0.11. This further supports our claim in the conclusion that performance may degrade when informative motifs are sparse or unavailable. We will include an updated section in the appendix with this ablation.

There is a lot of talk about the tradeoffs of "cheap" generators generating a TON of sequences and then having a good filtering set that can quickly weed out the bad sequences. Have the authors considered this?

We have not considered this filtering strategy. The goal of SpecMER is to design many high quality sequences, using k-mers as a form of filtering during generation, and we view our method as adjacent to this line of work. We will address all of your formatting concerns while making revisions. Thank you again for your feedback.

Thank you again for your constructive feedback. Please let us know if there is any remaining point you would like us to clarify or expand upon before the discussion period ends.