Steering Protein Language Models

We sincerely thank the reviewer for providing valuable feedback. We detail our response below point by point. Please kindly let us know whether you have any further concerns.

---

Q1: "the evaluation may exhibit a certain level of circular dependency. ... This could potentially be considered an attack on ESM-2. In this part of the experiment (Table 1), we also observe that ESM-2 shows the best performance. "

We thank the reviewer for raising the important issue of potential circular dependency in our evaluation. Using ESM-2 both as the base generative model and as part of the downstream predictor could potentially introduce biases. However, we emphasize the following points to address this concern:

• While using ESM-2 as a base model achieves superior performance in Table 1 to other base models, by comparing the generation performance with Action Steering and without Activation Steering (Original Model), we can we can observe a significant performance gain. This validates that improvements are attributable to the steering mechanism rather than inherent biases in ESM-2.

• Besides, our proposed Activation Steering method demonstrates consistent and significant improvements over both Fine-tuning and the Original Model across all three base models (ESM-2, ESM-3, and ProLLaMA). This cross-model consistency suggests the improvements stem from the method itself rather than inherent biases in ESM-2.

---

Q2: " It is necessary for the paper to include metrics that assess the authenticity of the generated sequences, such as pLDDT and sequence likelihood."

To address the concern about assessing the authenticity of generated sequences, we incorporate the pLDDT metric, evaluated using ESMFold, to measure the structural integrity of both initial and optimized protein sequences. The results, presented in the tables below, demonstrate that the pLDDTs remain consistent before and after applying activation steering. This consistency confirms that our method effectively guides protein generation towards desired functionalities while maintaining the authenticity of the sequences

Table 1: pLDDT for protein generation.

Table 2 pLDDT for protein optimization regarding thermostability.

Table 3 pLDDT for protein optimization regarding solubility.

---

Q3: "I am curious whether the paper's method performs well for modeling more niche properties... as it fundamentally relies on the internal knowledge of PLM."

• We appreciate the reviewer's interest in the applicability of our method to more niche properties. We would first note that the challenge with niche properties is the scarcity of labeled data, which can hinder the training of effective predictors to estimate the performance.

• We conducted an additional experiment focusing on the niche property of hydrolysis activity at 30°C for polyethylene terephthalate (PET). We utilized the dataset from (Seo et al. 2025), comprising 184 annotated samples.

  • To estimate the performance, we train a predictor on this dataset, achieving a Pearson correlation coefficient (R) of 0.57 in 5-fold cross-validation, indicating a reasonable prediction capability under data constraints.
  • We design optimization tasks of varying difficulty:
    • Medium Difficulty Task: Optimizing 69 samples with an initial average predicted activity of 1.61.
    • Hard Difficulty Task: Optimizing 67 samples with an initial average predicted activity of 175.62.

• The results post-optimization using our ASPO method combined with ESM3 are as follows:

• These results demonstrate that our ASPO method can effectively optimize even niche protein properties, such as hydrolysis activity in PET, underlining the versatility and potential of our method in broader protein design applications.

Reference: Seo, Hogyun, et al. Landscape profiling of PET depolymerases using a natural sequence cluster framework. Science 2025.

Thank you for your thoughtful and positive feedback. We have provided a detailed explanation for your concerns as follows. Please feel free to let us know if you have any additional concerns or questions.

W2: Assumptions .. continuity, linearity...Theoretical grounding

• Our method is based on two hypotheses: the linear representation hypothesis and the superposition hypothesis (T. Adly 2024). The linear representation hypothesis suggests that neural networks encode meaningful concepts as directions in their activation spaces, while the superposition hypothesis extends this by suggesting that networks utilize almost-orthogonal directions in high-dimensional spaces to represent features, embodying properties of additivity and homogeneity. These hypotheses underpin the design of many existing algorithms.

Ref: T. Adly. Scaling monosemanticity: Extracting interpretable features from Claude 3 sonnet. Anthropic, 2024.

• However, the theoretical validation of these hypotheses is ongoing. As such, the theoretical grounding of our method remains a subject for future research.

M1: the assumption .. only true if dataset has large numbers of samples

• Our method assumes that the pretrained PLM has already encapsulated comprehensive knowledge of protein properties, given its pretraining on a massive number of both naturally occurring and synthetically designed protein sequences. This extensive pretraining dataset supports our assumption that the PLM possesses a universal understanding of various protein properties, making it suitable for steering the PLM to explore specific properties.

• However, we acknowledge that if the PLM lacks prior knowledge of a target property, our method may not be effective, which is a limitation.

• Regarding the reviewer's point on fine-tuning, it is important to clarify that FT also relies on the pretrained model having some foundational knowledge of the desired properties. Without this, FT on a small dataset risks overfitting and poor generalization, similar to the limitations faced by our proposed method.

M2: unsure whether just adding a vector

• We appreciate the reviewer's concern regarding the potential risks of modifying latent representations. Indeed, adding arbitrary vectors can disrupt model outputs, akin to adversarial attacks. However, our method carefully defines the steering vector to ensure meaningful modifications aligned with desired properties.

• To illustrate, consider a simple case with a positive sample ($z_p$) and a negative sample ($z_n$). To shift $z_n$ towards $z_p$, the intuitive direction for modification is $v=z_p-z_n$. Extending this, if we have sets of positive and negative samples, the steering vector can be defined as the mean difference between these sets, aligning changes with the observed data distribution.

• Our method assumes that the latent space adheres to the linear representation hypothesis and superposition hypothesis, allowing these mean-based modifications to navigate towards desired properties effectively. Empirical results confirm that this method in steering the latent representation leads to the desired enhancements.

M3: not strictly true that adding the steering vector .. move it towards the desired property

• The effectiveness of the proposed method relies on the linear representation hypothesis and superposition hypothesis. The thought experiment designed by the reviewer, where the latent space exhibits non-linear characteristics, poses a challenge to our assumptions. In this case, the proposed method does not work.

M4&W3: Concerns about the magnitude .. reflecting relatedness to the property

• Normalization techniques like LayerNorm and RMSNorm in transformers constrain vector magnitudes, ensuring they don’t scale infinitely. This addresses the counterexample.

• Additionally, the magnitude is indicative of the importance of the token. Our relatedness score takes the magnitude of the token representations into consideration, following the attention mechanisms in transformers, which compute attention scores via dot products between queries and tokens.

M5&W4: maintain unrelated properties

• We evaluated both thermostability and solubility to ensure our method maintains unrelated properties during optimization. Due to length limitations, we only present solubility experiments. As shown in the following tables, steering for solubility has minimal impact on the unrelated property thermostability.

Tab: Protein generation

Tab: Protein optimization

We appreciate very much your constructive comments on our paper. We have provided a detailed explanation for your questions as follows. Please feel free to let us know if you have any additional concerns or questions.

---

Q1: "How do you deal with multi-property optimization?"

Thank you for your insightful question regarding our method for multi-property optimization..

• We propose a simple solution to perform steering on multiple properties by computing a composite steering vector that incorporates the steering vectors of individual properties. Specifically, if $v_\ell^{therm}$ represents the steering vector for thermostability and $v^{sol}_\ell$ for solubility, the combined steering vector used for multi-property optimization is given by

$v_\ell = v_\ell^{therm} + v^{sol}_{\ell}$.

• The remaining settings are the same as single property optimization.
• We empirically test this method in scenarios involving both protein generation and protein optimization. The results, as detailed in the tables below, demonstrate that our method effectively improves multiple properties, albeit with a slight trade-off compared to optimizing a single property.
• We plan to further refine our method to better manage these trade-offs and explore more sophisticated methods for steering vector combination and weighting for multi-property optimization in future work.

Table 1: Multi-Property Steering for Protein Generation

Table 2: Multi-Property Steering for Protein Optimization on Thermostability Medium Difficulty Task

Table 3: Multi-Property Steering for Protein Optimization on Thermostability Hard Difficulty Task

---

Q2: "In the ablation studies, it's better to show the trend of both properties and basic protein generation qualities with respect to the ablated variables."

Thank you for your valuable suggestion. We include trends for diversity, distance to the initial set, and distance tothe high fitness set in the following tables. and will update them in the corresponding figures. We will also reflect these trends in the revised figures to provide a clearer visualization of how the ablated variables impact both properties and basic protein generation qualities.

Table 4: Sensitivity to number of samples for steering vectors extraction in protein thermostability optimization (Fig. 4(a))

Table 5: Sensitivity of $\alpha$ in protein thermostability optimization (Fig. 6(a))

---

Q3: "the worse fine-tuning results might be due to insufficient data and large numbers of learnable parameters. Do you use parameter-efficient fine-tuning, like LoRA?"

• We appreciate your comment regarding parameter efficiency in fine-tuning. Indeed, we employ LoRA, specifically with a rank of 8, which we found optimal in our experiments after evaluating various ranks [2, 4, 8, 12, 16]. The rank of 8 outperformed others, with rank 4 being the next best. Our choice of hyperparameter alpha is 16.

We sincerely thank the reviewer for providing valuable feedback. We detail our response below point by point.

W1: the ESM3 model ... generate a full sequence from all-mask states

We conduct experiments on full sequence generation using ESM3, maintaining the same settings as described in Section 4.1. The results are presented in the table below, demonstrating that Activation Steering significantly outperforms the baseline methods, thereby confirming its effectiveness.

W2: more details about experiments

• The number of iterations T and the value of K
  • Protein optimization: thermostability: T=8, K=4; solubility: T=4, K=2; GFP: T=4, K=2
• Length distribution of generated sequences
  • Protein generation
    • thermostability: 60~256, mean: 194.1
    • solubility: 47~256, mean: 168.5
  • Protein optimization
    • thermostability
      • medium difficulty: 60~256, mean: 183.9
      • hard: 102-256, mean: 208.2
    • solubility
      • medium: 71~256, mean: 180.4
      • hard: 47~253, mean: 158.9
    • GFP: length of all sequences is 237

W3: further explain the lines 189-194

Our method first involves computing a relatedness score for each token to identify which should be mutated, as discussed prior to the mentioned lines. In lines 189-194, we propose to determine the layer for computing the relatedness score as the one with the highest validation accuracy. This ensures that the most informative layer are utilized for token selection.

After determining the tokens for mutation, we replace these tokens with a mask token. Importantly, activation steering is applied during inference across all layers, except the input layer, to steer the model's prediction on the masked tokens towards the desired property.

We will revise lines 189-194 to enhance clarity on these points.

W4: Missing discussion of PPLM

Thank you for pointing out the omission of PPLM. PPLM indeed pioneers the concept of steering by modifying key-value pairs in the model's attention mechanism, guided by gradients from an attribute model. In contrast, our method employs a simpler activation steering approach that directly manipulates activations and does not require training an additional attribute model or updating steering vectors.

We will include discussion of PPLM in the related work to highlight these distinctions.

Q1: Could the performance of steering be further boosted by enlarging the model scale

• We conducted experiments using ESM2-3B for protein sequence generation, maintaining the same settings as in Sec 4.1. The results are summarized in the table below.
  • Compared to ESM2-650M, the proposed method Activation Steering shows similar performance in thermostability and significantly improves solubility.
  • In contrast, fine-tuning performance decreases with larger ESM2, likely due to the need for more data to achieve optimal results.

Q2: the performance when the number of samples is further reduced

• We appreciate the reviewer's interest in the performance of AR-PLM with fewer samples. To address this, we conducted additional experiments using ProLLaMA with sample sizes ranging from 1 to 10. Each configuration was tested 10 times to mitigate randomness. The results, summarized in the table below, indicate that the optimal number of samples varies depending on the property being optimized. For thermostability, the best performance occurs with 8 samples, while for solubility, it peaks at 3 samples. Notably, the performance does not consistently improve with fewer samples; the lowest number of samples (1 sample) did not yield the best results.
• This suggests that while reducing the number of samples can sometimes enhance performance, likely by focusing the generation on a narrower subcluster of proteins with desired properties, there is a trade-off in terms of robustness. Performance becomes less predictable and can vary significantly depending on the specific samples used to compute steering vectors.
• In conclusion, while fewer samples can sometimes be beneficial, the optimal number of samples depends on the specific application and desired property, balancing performance with robustness.

Q3: compatible with other protein steering or optimization methods?

In this paper, we focus on studying steering for PLM and we apply PLM steering exclusively within the proposed ASPO method for protein optimization, positioning ASPO as a competitor rather than complementary to existing methods. Integration with other protein optimization strategies remains an interesting direction for future research.