Implicitly Guided Design with PropEn: Match your Data to Follow the Gradient

[Setting $\Delta_x$ and $\Delta_y$]

The choice of parameters $\Delta_x$ and $\Delta_y$ should be informed by the specific application. For example, in antibody design, domain experts recommend not using thresholds above a Levenshtein distance of 8, as such differences are considered biologically irrelevant. Similarly, for $\Delta_y$, knowing that noise in binding measurements can be up to 0.3, we chose 0.5 to ensure proper matching.

When faced with a new dataset of $n$ unique data points, a good initial approach is to use the standard deviation of pairwise distances for $x$, and the mean or median for property $y$:

• $\Delta_x \lt \sigma_d = \sqrt{\frac{2}{n(n-1)} \sum_{1 \leq i < j \leq n} \left( d(x_i, x_j) - \frac{2}{n(n-1)} \sum_{1 \leq k < l \leq n} d(x_k, x_l) \right)^2}$

• $\Delta_y \lt \frac{2}{n(n-1)} \sum_{1 \leq i < j \leq n} d(x_i, x_j)$

[How g(.) influences the calibration/selection of d_y] When the $g(.)$ is non-smooth, or we have a very few datapoints, this certainly influences the choice of $\Delta_y$, in this case we opt for as small steps as possible since this will create many pairs which should give sense of the direction of the gradient around the sparse regions. In the attached PDF, we provide an additional ablation study on how threshold choices affect the number of training pairs, showing that sufficiently large thresholds include all unique training points. For performance impact, please refer to the ablation study in Figure 4.

[Multiple-Property Optimization]

We agree that multiple-property optimization is crucial in real-world applications, as we have experienced ourselves. In such scenarios, we compute a multivariate rank over all properties of interest and optimize for that single score with PropEn. The multivariate score can be computed using the Hypervolume [1] or the CDF indicator, as suggested by [2]. Both methods work effectively, and we provide results on in-silico multi-property antibody optimization in the attached PDF. We select four developability properties: number of liabilities, hyrophobicity, positive and negative charge. We compute a multi-property ranking over these 4 properties, resulting in a single, scalar score we will use in PropEn. We then train PropEn models to optimize for each of these properties individually, and an mPropEn which uses the multivariate ranking score. In the figure in the attached pdf, we observe that (1) for two orthogonal targets, mPropEn suggests designs on the Pareto Front, while the single-objective properties tend to suggest designs at the tails of the front as expected, and (2) taking all four objectives into account, the top 20% of designs at the Pareto front come from the variants of mPropEn. This validates our idea of combining PropEn with multi-objective indicators, and we plan to expand on other datasets and experiments in further analysis.

[1] Zitzler, Eckart, et al. "Performance assessment of multiobjective optimizers: An analysis and review." IEEE Transactions on evolutionary computation 7.2 (2003): 117-132.

[2] Park, Ji Won, et al. "BOtied: Multi-objective Bayesian optimization with tied multivariate ranks."ICML 2024.

[Seed Designs Selection and Flawed Designs]

Generally, seed designs come from the training data. In cases where they do not, our model's tendency to optimize locally ensures that out-of-distribution (OOD) seeds are mapped to the closest neighborhood in the training set, as shown in Figure 2 in the submission and figure 1 and 2 in the attached pdf. In contrast, other methods, such as explicit guidance, often fall off the manifold and suggest designs in regions unsupported by the training data.

We have found that starting with seeds at the distribution's edge, with the highest possible values, helps achieve significant improvements, as these seeds have the best chance of extrapolating. The attached PDF provides examples of seeds where PropEn achieved higher values than anything observed in the training distribution.

[Benchmarking Against Bayesian Optimization (BO)] Please see our general response.

[Connection to Diffusion Models] Please see our general response.

[Neighborhoods in L2 Ball]

The assumption regarding neighborhoods in L2 balls was made solely for theoretical purposes, to make the explanation more general and accessible. Neighborhoods in L2 balls naturally connect to training with Mean Squared Error (MSE) loss. However, the methodology is versatile and can be adapted to train with different losses; it can be shown that the same theory holds for Binary Cross-Entropy (BCE) loss as well. We are happy to expand on the proof in further discussion if the reviewer finds this interesting.

Similarity in $x$ should be defined based on the application, which is why we showcase different datasets. For antibodies, we used Levenshtein distance as an example. We believe this choice gives freedom to include more inductive biases from domain experts.

[Sensitivity of Matched Reconstruction to Sampling Choice]

There is no sampling choice made in PropEn; we keep all the possible pairs from the training data. This approach ensures that our method is comprehensive and does not rely on arbitrary sampling decisions.

[Regularization to Prevent Overfitting]

We have identified two ways to avoid overfitting:

1. Mixup technique: Including a reconstruction loss to the original sample in the training loss (the mixup variant in experiments) helps prevent overfitting. Due to the one-to-many matching scheme, we cannot perfectly reconstruct a single example, but rather learn how to interpolate between samples.

2. Property Value Inclusion: Incorporating the value of the property in reconstruction controls the gradient steps (the xy2xy variant).

The theoretical background on why this works is included in the supplementary material, Section A.4, "Understanding the Matched Reconstruction Objective."

[Extending Matching to Other Models] Please see our general response.

[Comparison to Active Learning] Please see our general response.

[Evaluating Target Properties on Neighbouring Samples]

Could you please elaborate on what you mean by "evaluating the properties on neighbouring samples"? We are not entirely sure about the question, but we hope this explanation helps: The matching is based on observed or measured values of properties for each example; we do not use any predictor to obtain them. For the toy example, the property was computed using a KDE estimator. For the airfoils, lift and drag coefficients were obtained by Neural Foil, and for the antibodies, binding affinity was measured through wet lab experiments.

[Complexity and Diversity of Examples]

We made an effort to demonstrate that this method works across different domains, neural network architectures, and tasks. Typically, machine learning papers follow a standard benchmark within a single application domain. Our experiments cover toy, engineering, and biology applications, showcasing a broader diversity. Additionally, we include results from wet-lab experiments, which cost $20,000 and take several months to complete. We hope the reviewer will reconsider and examine our manuscript with attention to these aspects.

I have read the comments and am satisfied with the clarifications and additional results. I will raise my score to borderline accept

[Details on Training and Matched Datasets] We add the following table in the supplement:

[Details on Models and the choice of ResNet]

We missed mentioning that the ResNet blocks in our network are non-convolutional. PropEn for antibodies uses fully connected layers with residual connections, one-hot encoding for input sequences, and layer normalization and GELU activation. This choice is tailored to the specific demands of antibody sequence analysis.

[Airfoil Optimization Baseline]

We added a diffusion baseline for the airfoil experiment following the implementation from [1]. The model is a 3-layer MLP with 128 units per layer, sinusoidal time and input embeddings, and GELU activations. We conducted a grid search over batch size, hidden layer size, learning rate, and time steps and repeated the experiment five times for the best parameters. Results for T \in \{ 5, 15, 50\}\ show that the diffusion model optimizes almost all hold-out designs, however with minor improvements in the lift-drag ratio. For the final manuscript, we'll explore more advanced diffusion models, particularly from reference [2].

1. tiny-diffusion
2. Diffusion-based 2DAirfoil Generation

[Statistical Analysis for Results in Section 3.2]

The expense of wet-lab experiments constrains us, limiting the number of tested examples. The overall cost for the results presented was approximately $20,000. State-of-the-art methods like Walk-Jump and Lambo include results from a single target and seed, whereas we expand to 4 targets and 8 seeds.

As requested, we add statistical significance tests using Fisher's Exact test and Chi-Square tests.

Statistical Significance Tests for Table 3: Binding rate per method;

Statistical Significance Tests for Table 4: Number of designs improving over seed per method;

[Toy Exp; choice of property]

We propose a setup where the property and likelihood are disentangled, retaining the shape of 8 Gaussians but computing a property increasing in value in an anticlock-wise direction. Please see the attached PDF for results.

[Numbers in Table 3]

We only reference the source publications and we independently run each of them our wet-lab experiments. This is why the numbers do not appear in their corresponding publications.

[Overparameterized Model Assumption] This is a general, common assumption in deep learning models. We will add a discussion on the overparameterized model; essentially, it is a standard assumption that has been theoretically [1, 2] and empirically explored in many prior works on deep learning. Overparameterization refers to the practice of using models with more parameters than the number of data points or the complexity strictly required to fit the training data. This assumption might seem counterintuitive initially, as it suggests using models that are larger than necessary. However, several compelling arguments justify this approach, highlighting its benefits for model performance, generalization, and learning dynamics.

[1] Bengio, Y. et al. "Representation learning: A review and new perspectives." IEEE transactions on pattern analysis and machine intelligence". [2] Allen-Zhu et al "A convergence theory for deep learning via over-parameterization." ICML 2019.

[Response to minor comments]

[Mixup/Mix] This version of PropEn, similar to the original Mixup you referenced, interpolates between examples. If this is confusing, we can consider renaming it.

[Uniqueness and Novelty]

• Uniqueness: #unique designs divided by #gen designs.
• Novelty: #designs proposed by the method that don't appear in the training data, divided by the #gen designs.

[How Were the Initial Airfoil Designs Chosen?] Random holdout dataset.

[Figure 4] When varying $\Delta_x$, $\Delta_y$ is set to 1, and vice versa.

[Figure 5] The size corresponds to the number of designs for that seed.

[Figure 8] Explicit guidance tends to fall off the manifold when many optimization steps are taken or seeds are OOD.