We thank the reviewer for their thorough reading of our work and for the positive feedback. We appreciate the reviewer’s suggestions on improving the mathematical notation, all of which have been incorporated in blue in the revised manuscript. Additionally, we have removed the claim regarding the absence of hyperparameters.

Regarding the Questions:

1. On likelihood base models and OOD Detection:
   Recent work [1] has shown that, for normalizing flows, the number of singular values of the Jacobian matrix exceeding a certain threshold can help address the OOD paradox (i.e., spuriously assigning a high likelihood to OOD data samples). We believe it is a promising direction to investigate whether ScaLES could also be used in this context to improve OOD detection for likelihood-based models.

2. On the Expression Dataset and using larger $\lambda$:
   For the expression dataset, the (black box) problem is easier compared with the other benchmarks, which is why we think the regularization is less helpful. We believe that the gradient of the acquisition function is not as noisy as in the other datasets, and there are fewer gains in adding regularization. We note that in general we did not perform a through grid search to find the optimal $\lambda$ for each problem and it is possible that for some cases using $\lambda>5$ would be beneficial.

[1] Kamkari et al., “A Geometric Explanation of the Likelihood OOD Detection Paradox,” ICML 2024.

We thank the reviewer for their thoughtful comments.

1. Likelihood and OOD Data

We would like to clarify that we did not claim OOD regions will always have lower density, but rather that OOD regions are more likely to have lower density—a claim quantitatively supported in Table 2. As noted in our original submission, likelihood consistently shows a strong correlation with being in-distribution, defined by decoding into valid objects. This is evidenced by the high AUROC (0.92 on average and always above 0.75) values observed across the various decoders we analyzed.

It is also important to note that Nalisnick et al.'s analysis focused on images, which involve a different data modality (continuous images versus discrete sequences) and applied a different definition of OOD, specifically based on images from different datasets rather than latent vectors and the concept of validity. A possible explanation for the differences between our findings and those of Nalisnick et al. is that our definition of validity more effectively distinguishes in-distribution data from out-of-distribution data compared to the approach used in their analysis of images from different datasets.

2. Interest in OOD Data Points

While it is true that OOD data points can sometimes be of interest, the key issue is not their potential value but whether there exists a practical algorithm to systematically and efficiently identify such points. For LSO, our analysis (i.e., the poor performance of LSO (L-BFGS) method) and previous work ([1] and [2]) suggest that no such algorithm currently exists. To substantiate this claim, we include below relevant excerpts from the cited literature in our introduction:

“Although in principle optimization can be performed over all of Z, it has been widely observed that optimizing outside of the feasible region tends to give poor results, yielding samples that are low-quality, or even invalid (e.g., invalid molecular strings, non-grammatical sentences); therefore, all LSO methods known to us employ some sort of measure to restrict the optimization to near or within the feasible region.”[1]

“Automatic Chemical Design possesses a deficiency in so far as it fails to generate a high proportion of valid molecular structures. The authors hypothesize that molecules selected by Bayesian optimization lie in ‘dead regions’ of the latent space far away from any data that the VAE has seen in training, yielding invalid structures when decoded.”[2]

We believe these examples, drawn from highly cited papers written by leading researchers in the field, provide compelling evidence that while OOD data points might occasionally be valuable, there are no practical, systematic methods to reliably identify them. We will be happy to further emphasize this point in the revised manuscript, to make it extra clear.

3. Empirical Improvements and OOD Regions

Finally, regarding the question of whether the empirical improvements we observe might stem from exploring OOD regions: we do not believe this to be the case. The following points support our reasoning:

• The analysis in Table 2 demonstrates that LES consistently achieves higher values in regions of the latent space that decode into valid objects, which we and others ([1] and [2]) define as in-distribution.
• Across 30 experiments, the proportion of valid solutions generated during optimization with LES is always higher than without LES. (Table 10, LES is always higher than LSO (GA))

Given these observations, we find it highly unlikely that LES explores OOD regions, as these would typically not decode into valid objects (per [1] and [2]).

References

[1] Tripp, A., Daxberger, E. and Hernández-Lobato, J.M., 2020. Sample-efficient optimization in the latent space of deep generative models via weighted retraining. Advances in Neural Information Processing Systems, 33, pp.11259-11272.

[2] Griffiths, R.R. and Hernández-Lobato, J.M., 2020. Constrained Bayesian optimization for automatic chemical design using variational autoencoders. Chemical Science, 11(2), pp.577-586.

We thank the reviewer for their careful reading of our work, and we also appreciate their suggestions for improving our manuscript. We follow the original structure of the review in our response.

CPA Assumption

We thank the reviewer for their insightful comment and for highlighting the relevant reference showing that the CPA assumption is not strictly required to derive the change-of-variable formula. We appreciate the opportunity to clarify why the CPA assumption is important in our work.

1. Our derivation in Equation (10) enables us to compute the gradient of ScaLES as a function of the softmax probabilities and the decoder’s derivatives (the slope matrices $A_\omega$) in closed form. This approach reduces the computational burden compared to differentiating through $S(z)$ using automatic differentiation.

2. Calculating the derivative of ScaLES without the CPA assumption, as described by Ben-Israel, would require (via Jacobi's formula) computing (denoting the decoder as $\boldsymbol{G}(z)=\text{Softmax}(\boldsymbol{L}(z))$):
   $\frac{dS(z)}{dz_i} = \text{Trace}((J\boldsymbol{G}^T J\boldsymbol{G})^{-1} \frac{dJ\boldsymbol{G}^T J\boldsymbol{G}}{dz_i})$

   The term $\frac{dJ\boldsymbol{G}^T J\boldsymbol{G}}{dz_i}$ involves second-order derivatives of the decoder network, resulting in a significant computational burden that scales quadratically with the latent dimension in gradient evaluations. However, representing the decoder as a CPA implies that the second-order derivative of the logits matrix $\boldsymbol{L}$ is always zero, which alleviates the need to compute these derivatives.

3. Lastly, the CPA assumption provides intuition by partitioning the output space into regions, each associated with a distinct contraction or dilation, affecting the likelihood of it's image

The reviewer is correct that this important clarification was missing in the original manuscript. We have revised the manuscript to address this point (l.264–269).

On Scalability

We appreciate the reviewer’s suggestion to clarify the term "scalable," and we agree with their assessment. The reviewer is correct that while the calculation of the determinant may not constitute a computational burden in the VAEs we study (which are comparable in size to VAEs used in real applications, e.g., [1]), it will not scale gracefully to much larger VAEs. Reflecting on the reviewer’s comments, we have:

• Removed the term "scalable" from the manuscript.
• Emphasized the computational complexity of our method in the contributions (l. 92).

[1] Truong, G.B., et al, 2024. Discovery of Vascular Endothelial Growth Factor Receptor 2 Inhibitors Employing Junction Tree Variational Autoencoder with Bayesian Optimization and Gradient Ascent. ACS Omega.

Validity of ScaLES Heuristic

We fully agree with the reviewer’s observation that the success of ScaLES is closely linked to the performance of the decoder. This is a point we emphasized in lines 203–204. In the revised manuscript, we:

• Removed the term "general-purpose."
• Included a more detailed discussion of this under the limitations of Theorem 5 (l.284–286).

Please also see our response to R3 for a discussion regarding the 'Interest in OOD Data Points'.

Overclaims in Writing

We thank the reviewer for raising concerns about our claims. In response, we have removed the terms exact, general-purpose, theoretically motivated, and scalable from the manuscript.

Baseline Method Descriptions

We have added a description of the Bayesian uncertainty score in Appendix D (l.1045–1052).

Questions

1. What is the purpose of the CPA approximation?

   As explained in our response above, the CPA approximation:

   • (1) Enables closed-form expressions for the gradient of ScaLES.
   • (2) Avoids the need to compute the Hessian of the decoder.
   • (3) Provides a geometric interpretation of the decoder function as a map that partitions the latent space into regions, before applying the softmax function.

2. What is the meaning of "scalability" in reference to ScaLES?

   We have removed the term "scalable" from our manuscript.

3. Is the latent dimension of the generative models considered in this work similar to what one would expect to use in more practical settings for LSO?

   To the best of our knowledge, the VAE models considered in this study have latent dimensions comparable to those used in practical applications. We have provided a citation backing up this claim in our response above (On Scalability).

4. Lastly, I am curious if the authors have considered alternative instantiations of the ScaLES heuristic?

   The suggestion of using normalizing flows as an alternative is intriguing and certainly worth further exploration. In this study, we focus on methods that do not require additional training and can therefore be easily integrated into existing pipelines (l.85-86).

We thank the reviewer for their careful reading of our work and for identifying typos, which have been corrected in the revised manuscript. Indeed, "ScaLES/Uncertainty" describes the ratio and not another method.

Likelihood Clarification

First, we would like to clarify that the likelihood used in our approach corresponds to that of a sequence of L categorical random variables, each with D categories.

Second, we wish to emphasize that ScaLES is not an estimator of p(X=x∣Z=z), but of $p_{X(Z)}(X = x(z))$. This distinction is depicted in Fig. 2 in the original manuscript, and we have added further mention of this fact in l.184-185. Specifically:

• ScaLES does not assume a joint probabilistic model for X and Z.
• Instead, we consider X as a deterministic function of Z—the random variable—mapped through the decoder.

This is an important distinction, as many existing methods, such as the uncertainty method proposed by Notin et al., do assume that X follows the conditional distribution p(X=x|Z=z). Consequently:

• ScaLES describes a single likelihood function pX(Z)(X = x(z)) over the output space.
• p(X=x∣Z=z) defines a distinct likelihood function for each z.

Variational Inference and Likelihood

We would like to note that p(X=x|Z=z) is estimated through variational inference techniques, which are themselves approximate (e.g., training neural networks). Therefore, it holds no theoretical guarantees under any realistic assumptions. Rather, similar to ScaLES, it assumes that the decoder fits the data well, serving as a good-enough approximation to the true distribution.

New Likelihood-Based Score

Inspired by the reviewer’s insightful comments, we have incorporated a new score into our evaluation. This score quantifies the likelihood of the most probable output for a given latent vector z under p(X=x|Z=z), formally defined as:

$L(z) = max_x p(X=x|Z=z)$.

Notably, to the best of our knowledge, employing this score as a constraint in LSO has not been explored in prior work.

Comparative Evaluation with Likelihood Score

To facilitate a direct comparison between these two scores, we reproduced our entire evaluation section (Tables 1–9) to include the likelihood score. We are happy to report that our findings indicate that, in general, ScaLES outperforms the likelihood score as a scoring metric. Specifically:

• For the average rank of the top 20 solutions, ScaLES achieved the lowest average rank of 1.83 (lower is better) compared to 2.8 for the likelihood approach. (l.480-481)
• For the top 1 solution, ScaLES obtained the lowest average rank of 1.97, outperforming the likelihood approach, which achieved an average rank of 2.93. (l.1024-1025)

Addressing Transformer Network Study

Lastly, we thank the reviewer for the suggestion to study Transformer networks. We would like to emphasize (Table 2 in the original manuscript) that we study 10 decoders with a Transformer architecture. Our study is restricted to VAEs, as all past literature we are aware of conducted LSO using a VAE.

We thank the reviewer for their thoughtful critique of our work. We are pleased to note that the reviewer does not dispute our conclusion that LES outperforms the likelihood score, as supported by the results presented in Tables 2, 3, 8, and 10. Below, we summarize these findings:

• Table 2: LES outperforms the likelihood score in identifying OOD data points, achieving an average AUROC of 0.93 compared to 0.91 for the likelihood score (l.377).

• Table 3: LES regularization demonstrates superior optimization results for the top 20 solutions:

  • Average ranking (lower is better): LES achieves a value of 1.83, compared to 2.8 for the likelihood score.
  • Number of times a solution is within 1 standard deviation of the best solution (higher is better): LES achieves 21 instances, compared to 14 for the likelihood score.

• Table 8: LES regularization improves optimization results for the best solution found:

  • Average ranking (lower is better): LES achieves 1.97, compared to 2.93 for the likelihood score.
  • Number of times a solution is within 1 standard deviation of the best solution (higher is better): LES achieves 18 instances, compared to 16 for the likelihood score.

• Table 10: LES regularization delivers a higher percentage of valid solutions across all tasks, with an average of 0.61 compared to 0.58 for the likelihood score.

We would like to emphasize that our manuscript explicitly recognizes the likelihood score as “a more computationally efficient alternative, which comes with some performance trade-offs” (lines 308-309).

We also appreciate that the reviewer does not dispute the conceptual differences between the two approaches, as highlighted in both our previous response and the work of Nalisnick et al., cited by Reviewer 3. To reinforce this point, we include relevant excerpts from Nalisnick et al.:

“The VAE and many other generative models are defined as a joint distribution between the observed and latent variables. However, another path forward is to perform a change of variables. In this case x and z are one and the same, and there is no longer any notion of a product space X × Z.”

“The change of variables formula is a powerful tool for generative modeling as it allows us to define a distribution p(x) entirely in terms of an auxiliary distribution p(z), which we are free to choose, and f.”

Beyond the quantitative and conceptual differences highlighted above, we conducted an additional experiment to further demonstrate the advantages of LES in recovering the true density.

In this experiment, we consider a d-dimensional (d=25,56,75 and 256 to match the latent space dimensions in our study) Gaussian vector (z) transformed into a vector of “probabilities” using the softmax transformation. For each data point, we calculated LES and the likelihood score, along with the true density of X under the softmax transformation. To visualize the differences, we sampled evenly spaced data points between -20 and 20 along the first dimension of z, while sampling the other dimensions from a Gaussian distribution with a standard deviation of 0.1. These results, presented in Appendix C1 of the revised manuscript, clearly demonstrate that LES provides a more accurate estimate of the true density of X. In contrast, the likelihood score fails to capture the true density's correct structure.

To summarize, we have demonstrated the following differences between LES and the likelihood:

1. Mathematical difference: LES and the likelihood score are based on different formulas.
2. Conceptual difference: The two approaches make distinct probabilistic assumptions.
3. Large differences in density estimation: LES outperforms the likelihood score in a toy model setting.
4. Performance difference in LSO: LES demonstrates superior results when applied to LSO.

[1] Nalisnick, Eric, et al. "Do Deep Generative Models Know What They Don't Know?." International Conference on Learning Representations. 2019

Dear reviewer, did our response help in clarifying the difference?