Inversion-based Latent Bayesian Optimization

• [W1, Q1, L1] Discussion on the limitations and practical considerations of LBO.

  Good suggestion. While standard BO struggles with discrete data, LBO addresses this by mapping discrete data to continuous space. To bridge the gap between the discrete and continuous space, LBO utilizes a Variational AutoEncoder (VAE), the quality of which depends on the number of unlabeled data available. Thus, LBO requires a large amount of high-quality unlabeled data to train an effective VAE.

• [W2, Q2, L2] Realism and violated situation of assumptions in Proposition 1.

  Thank you for your valuable question. The Lipschitz continuity assumption for the objective function $f$ is common in Bayesian optimization [1-5] or global optimization [6]. Furthermore, we have shown that the distance $d_{\cal X}(\mathbf x, p_\theta(\mathbf z))$ can be reached to zero through our inversion method in Figure 12. These findings imply that our assumptions in Proposition 1 are highly realistic. Notably, the non-zero values of the distance are the main motivation of InvBO. If the assumptions are violated, GP prediction error increases, resulting in suboptimal optimization performance as we have already shown in Figure 8.

  • Reference

    [1] González, Javier, et al. "Batch Bayesian optimization via local penalization." Artificial intelligence and statistics. PMLR, 2016.

    [2] Scarlett, Jonathan. "Tight regret bounds for Bayesian optimization in one dimension." International Conference on Machine Learning. PMLR, 2018.

    [3] Hoang, Trong Nghia, et al. "Decentralized high-dimensional Bayesian optimization with factor graphs." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.

    [4] Kim, Jungtaek, and Seungjin Choi. "On local optimizers of acquisition functions in bayesian optimization." Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Part II. Springer International Publishing, 2021.

    [5] Lee, Seunghun, et al. "Advancing Bayesian optimization via learning correlated latent space." Advances in Neural Information Processing Systems 37 (2023).

    [6] Christodoulos A. Floudas and Panos M. Pardalos, editors. Encyclopedia of Optimization, Second Edition. Springer, 2009.

• [W3, Q5] Validation of INV and PAS on various LBO methods.

  We have already provided the experimental results of applying InvBO over mentioned LBOs, TuRBO-L, and LOL-BO in Table 2 of Section D. Table 2 shows that applying our InvBO to previous trust region-based LBOs (TuRBO-L, LOLBO, CoBO) and non-trust region-based LBOs (LBO, W-LBO, PG-LBO) consistently improves the optimization performance in 8 tasks.

• [W4, Q3, L3] Theoretical grounding for PAS.

  Thank you for the good suggestion. We also have a deep interest in the theoretical grounding for PAS and recognize its significance. To provide a comprehensive theoretical analysis of PAS, we are actively conducting ongoing research in this area. We anticipate sharing these findings in future work.

• [W5, Q4] Analysis of performance improvement on DRD3 and Arithmetic expression tasks.

  In the DRD3 and arithmetic expression tasks, the dissimilarity between $\mathbf{x}$ and $p_\theta(\mathbf{z})$ in previous LBOs without InvBO is relatively small, resulting in less room for performance improvement from inversion compared to other tasks. We conducted additional experiments measuring normalized Levenshtein distance between $\mathbf x$ and $p_\theta(\mathbf z)$ with and without inversion on DRD3 and arithmetic expression tasks in Figure 6. in the provided PDF. Compared to Figure 12 in Section C which has a mean of dissimilarity of 0.3, the mean of dissimilarity of these two tasks is lower than 0.1.

• [W1] Importance of $d_{\cal X}$ bounding assumption in Proposition 1.

  For analytical convenience, we use the bounding assumption of the distance function. This is similar to the concept of image normalization, where pixel values are scaled to a specific range (often 0 to 1) to facilitate more effective processing and analysis. We would like to note that this scaling does not affect the generality or applicability of our findings, since the underlying principles remain unchanged regardless of the bound.

• [W2, L1] Appropriateness of Levenshtein distance for molecules.

  Thank you for the good suggestion. Since we evaluate the optimization problem in diverse domains, such as molecule and arithmetic expression tasks, we use a more general distance metric, Levenshtein distance, compared to Tanimoto similarity. Our InvBO can utilize any distance function, including Tanimoto similarity. We will include it in the final version.

• [W3] Assessment of InvBO on the Sample Efficiency Matters Benchmark.

  Thank you for the suggestion. We conduct additional experiments applying InvBO to LBO in the sample efficiency matters benchmark. We apply InvBO to the SELFIES VAE provided by the benchmark on 7 tasks. The experimental results are in the table below:

  	REINVENT SMILES	Graph GA Fragment	SELFEIS VAE - InvBO	SELFIES VAE - LBO
  amlodipine_mpo	0.635 $\pm$ 0.035	0.661 $\pm$ 0.020	0.594 $\pm$ 0.011	0.516 $\pm$ 0.005
  median2	0.276 $\pm$ 0.008	0.273 $\pm$ 0.009	0.236 $\pm$ 0.018	0.185 $\pm$ 0.001
  osimertinib_mpo	0.837 $\pm$ 0.009	0.831 $\pm$ 0.005	0.823 $\pm$ 0.007	0.765 $\pm$ 0.002
  perindopril_mpo	0.537 $\pm$ 0.016	0.538 $\pm$ 0.009	0.538 $\pm$ 0.004	0.429 $\pm$ 0.003
  ranolazine_mpo	0.760 $\pm$ 0.009	0.728 $\pm$ 0.012	0.762 $\pm$ 0.011	0.452 $\pm$ 0.025
  valsartan_smarts	0.179 $\pm$ 0.358	0.000 $\pm$ 0.000	0.003 $\pm$ 0.005	0.002 $\pm$ 0.003
  zaleplon_mpo	0.358 $\pm$ 0.062	0.346 $\pm$ 0.032	0.384 $\pm$ 0.008	0.206$\pm$ 0.015
  SUM	3.582	3.377	3.34	2.103
  Rank	1	2	3	23

  We re-rank the method provided in the sample efficiency matters benchmark on 7 tasks by AUC Top-10 from 5 independent runs. In the table, applying InvBO to SELFIES VAE achieves rank 3 on 7 tasks while vanilla LBO with SELFIES VAE achieves rank 23. These results demonstrate that LBO with InvBO is highly competitive with other non-BO baselines.

• [W4, Q2] Additional comparison to T-LBO and analysis of the interplay between Inversion and metric learning.

  We appreciate for valuable suggestion. We provide the optimization results of T-LBO with and without inversion on the arithmetic expression task in Figure 5 in the provided PDF. The metric learning proposed in T-LBO adjusts the latent space to be smooth, making it easier to fit the GP. Meanwhile, inversion constructs an aligned dataset without additional oracle calls, enabling the GP to correctly emulate the objective function. Although these are orthogonal approaches, metric learning helps reduce the $L_3$ constant in Proposition 1, and it is expected to produce a synergistic effect with InvBO. We will include it in the final version.

• [W5] Tanimoto similarity between $\mathbf x$ and $p_\theta(\mathbf z)$.

  Thank you for the suggestion. We additionally conducted experiments measuring the Tanimoto similarity between $\mathbf{x}$ and $p_\theta(\mathbf{z})$ in Figure 4 in the provided PDF. Figure 4 shows the Tanimoto similarity comparison results with and without inversion on the med2 and valt tasks. We used rdkit library to measure the Tanimoto similarity. From the figure, the inversion achieves a Tanimoto similarity of 1.0 for every iteration, while CoBO without inversion achieves about 0.8 Tanimoto similarities in both tasks.

• [W6, L2] Motivation for $\alpha$ normalization in PAS.

  We normalize the $\alpha$ value to ensure that it has the same influence relative to the anchor point’s objective score $y$. Without scaling, regions with high uncertainty may be overly influenced by the $\alpha$ value compared to the objective score $y$.

• [W7, L3] Providing clear reproduction instructions.

  We will publish our code, including clear reproduction instructions in the camera-ready version if the paper gets accepted.

• [Q1] Assessment of generated molecules against the pre-training dataset.

  	med2	valt
  Ratio of novel data (%)	100 $\pm$ 0.00	100 $\pm$ 0.00

  We additionally conducted experiments to assess whether the generated data are contained within the pre-training Guacamol benchmark dataset. The table above provides the ratio of generated data that is not contained in the pre-training dataset on the med2 and valt tasks across five runs. The table demonstrates that all generated data during the optimization process are not contained in the pre-training dataset.

• [Q3] Sparse GP approximation method used in InvBO.

  As shown in our codebase, we use Sparse Variational Gaussian Process (SVGP) for the sparse Gaussian process implementation. We will add this detail in our camera-ready version if the paper gets accepted.

• [Q4] Discrepancies between Figures 7 and 16.

  Good observation. Figures 7 and 16 are different because they are conducted on different splits. Figure 7 illustrates the Gaussian process fitting results on the test set, while Figure 16 shows the results on the training set.

• [Minor points in Weakness]

  We appreciate your detailed review. We will update the final version considering these minor points.

• [W1] Modifying the colors of Figure 1 for red-green color blind folks.

  Thank you for the suggestion. We will modify the colors of Figure 1 for red-green color-blind folks in our camera-ready version if the paper gets accepted.

• [W2] Additional comparison to GEGL.

  Thank you for the suggestion. We provide the experimental results of GEGL optimization on adip and osmb tasks in Figure 3 in the provided PDF. We include only a subset of baselines in Figure 3 to enhance the clarity of additional experimental results. While GEGL demonstrates superior optimization performance compared to Graph-GA, CoBO with InvBO still achieves higher optimization performance.

• [Q1] Future work of InvBO.

  LBO has become a promising approach for optimizing structured data such as molecules or proteins. However, research on multi-objective Bayesian optimization over latent space has not been fully explored [1, 2]. Here, we propose InvBO for single-objective Bayesian optimization over latent space, but we believe that the misalignment problem also occurs in multi-objective Bayesian optimization over latent space. We will explore the adaptation of InvBO to multi-objective Bayesian optimization over latent space.

  • Reference
    1. Stanton, Samuel, et al. "Accelerating bayesian optimization for biological sequence design with denoising autoencoders." International Conference on Machine Learning. PMLR, 2022.
    2. Gruver, Nate, et al. "Protein design with guided discrete diffusion." Advances in neural information processing systems 37 (2023).

• [W1] Methodological contribution of InvBO beyond LOLBO and CoBO.

  Our InvBo can be applied beyond trust region-based LBO methods (e.g., LOLBO and CoBO). We show that each component of our InvBO, such as Inversion and Potential-aware trust-region anchor selection (PAS), can be implemented with diverse BO approaches, as specified below:

  • Inversion.

    Our Inversion can be applied to any LBO method, including trust region-based LBO methods (e.g., LOLBO and CoBO). In Table 2 of Section D, we have already shown that the inversion has successfully been adopted to diverse LBO works with and without using trust region.

  • PAS.

    Our PAS can be extended to any trust region-based standard BO method (e.g., TurBO). To validate it, we conduct additional experiments applying PAS to TuRBO on standard BO tasks in Rover and Lunar and report the optimization results in Figure 2 of the provided PDF. These experimental results demonstrate that InvBO can be applied not only to TR-based LBO but also to standard TuRBO.

• [W2] Appropriateness of Lipschitz assumptions in Proposition 1.

  Thank you for your valuable question. In Proposition 1, we assumed the black box function $f$ and the composite function $f\circ p_\theta$ of the black box function and the decoder of VAE as a Lipschitz continuity function. Assuming the black box function or the objective function as the Lipschitz continuous function is common in Bayesian optimization [1-5] or global optimization [6].

  • Reference

    [1] González, Javier, et al. "Batch Bayesian optimization via local penalization." Artificial intelligence and statistics. PMLR, 2016.

    [2] Scarlett, Jonathan. "Tight regret bounds for Bayesian optimization in one dimension." International Conference on Machine Learning. PMLR, 2018.

    [3] Hoang, Trong Nghia, et al. "Decentralized high-dimensional Bayesian optimization with factor graphs." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.

    [4] Kim, Jungtaek, and Seungjin Choi. "On local optimizers of acquisition functions in bayesian optimization." Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2020, Ghent, Belgium, September 14–18, 2020, Proceedings, Part II. Springer International Publishing, 2021.

    [5] Lee, Seunghun, et al. "Advancing Bayesian optimization via learning correlated latent space." Advances in Neural Information Processing Systems 37 (2023).

    [6] Christodoulos A. Floudas and Panos M. Pardalos, editors. Encyclopedia of Optimization, Second Edition. Springer, 2009.

• [Q1] Applying PAS to TuRBO on standard BO benchmarks.

  Thank you for the suggestion. We provide the optimization performance of TuRBO and TuRBO with PAS in two standard BO benchmarks, Rover and Lunar. Our implementation is based on the codebase of TuRBO provided in the BoTorch tutorial and uses the same hyperparameters ($e.g.$, batch size, and number of initial data points) with TuRBO. The experimental results are reported in Figure 2 of the provided PDF. The figure shows that applying PAS on TuRBO consistently improves the optimization performance. These results demonstrate that PAS is also effective in standard BO benchmark tasks.

Thank you for your thoughtful consideration and revisiting your initial concerns. We will incorporate your valuable feedback in our final version.

• [W1] Motivation of InvBO.

  Other reviewers provided positive comments regarding the motivation of our paper as follows:

  Reviewer 8vpX: This paper identifies and addresses an overlooked issue in several latent space Bayesian optimization methods.

  Reviewer wVbQ: The way the authors motivated, defined, and applied InvBO is clear.

  Reviewer nFyz: The authors highlight a systematic problem present in all VAE-based Bayesian optimization architectures.

  Reviewer tYNa: The inversion method is a novel and principled way to address the important problem of misalignment between the input and latent spaces in LBO.

  Here we clarify the motivation behind our proposed InvBO, which consists of Inversion and PAS.

  • Motivation of Inversion.

  LBO suffers from the misalignment problem caused by the reconstruction error of the VAE. Previous works handle this problem with a recentering technique; however, Figure 3 shows that this requires additional oracle calls. This motivated us to design an inversion method, a solution to the misalignment problem that does not require any additional oracle calls.

  • Motivation of PAS.

  Most prior trust region-based approaches select the anchor as the current optimal point without considering the potential of the latent vectors within the trust region to improve optimization performance. This prompted us to design a novel anchor selection method that considers the potential of the latent vectors within the trust region.

• [W2, Q1] Evidence and analysis of misalignment problem.

  We have already presented evidence that the misalignment problem certainly degrades optimization performance. Figure 12 in Section C shows the discrepancy between $\mathbf{x}$ and $p_\theta(\mathbf{z})$. As shown in Figure 7 from the main paper, this discrepancy leads to the misalignment problem. Furthermore, Figure 8 in the main paper shows that optimization performance is significantly lower with a misalignment problem (yellow line) compared to when the problem is addressed by inversion (blue line).

• [W3, Q6] Diversity of experimental domains and general optimization ability of InvBO.

  We already measure the performance of our InvBO on diverse domains such as molecule domains (e.g., Guacamol and DRD3 tasks) and an arithmetic expression fitting task. Figures 4 and 5 in the main paper demonstrate the general optimization ability of our InvBO across various domains and settings.

• [Q2] Exploration capability of InvBO.

  InvBO does not make exploitation-centric decisions. In Figure 1 of the provided PDF, we conduct additional experiments measuring the number of unique data searched in each iteration of CoBO with and without InvBO. Figure 1 demonstrates that InvBO does not lose exploration capability compared to CoBO. On the other hand, Figure 4 in the main paper shows that applying InvBO enhances the exploitation ability. These results indicate that InvBO makes decisions while balancing exploitation and exploration.

• [Q3] Clarification on the appropriateness of the term 'Inversion'.

  ‘Inversion’ is broadly used to refer to the reverse process of generation and has been widely applied to generative models such as GANs and Diffusion models [1-5]. As mentioned in Section 2.2 of the paper, ‘Inversion’ is a process of finding a latent code that generates the original data $\mathbf{x}$ through a generator $G$. Without loss of generality, we also use the term `Inversion' in the same sense with decoder $p_\theta$ and formally define it in Equation (4).

  • Reference
    1. Xia, Weihao, et al. "Gan inversion: A survey." TPAMI, 2022.
    2. Zhu, Jiapeng, et al. "In-domain gan inversion for real image editing." ECCV, 2020.
    3. Wang, Tengfei, et al. "High-fidelity gan inversion for image attribute editing." CVPR, 2022.
    4. Xu, Yiran, et al. "In-N-Out: Faithful 3D GAN Inversion with Volumetric Decomposition for Face Editing." CVPR, 2024.
    5. Gal, Rinon, et al. "An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion." ICLR, 2023

• [Q4, Q7] Details of the dimensionality of latent space and the Variational Autoencoder (VAE) used in experiments.

  For a fair comparison, we use the same dimensionality of latent space and VAE following baseline works. As we mentioned in the main paper, we use SELFIES VAE [1] for the de novo molecule design tasks (e.g., Guacamol and DRD3) and Grammar VAE [2] for the arithmetic expression fitting task.

  • Paper
    1. Maus, Natalie, et al. "Local latent space Bayesian optimization over structured inputs." NeurIPS, 2022.
    2. Kusner, Matt J., et al. "Grammar variational autoencoder." ICML, 2017.

• [Q5] Theoretical analysis of InvBO.

  In Proposition 1, we theoretically show that inversion plays a crucial role in minimizing the upper bound of the GP prediction error within the trust region. Figure 9 in the main paper shows the experimental results of GP prediction error within the trust region (left) and the corresponding optimization results (right). These results demonstrate that the inversion method minimizes the GP prediction error and results in the improvement of the optimization process.

• [Q8] Detailed description of Figure 3.

  The left figure illustrates the number of oracle calls made by the acquisition function and recentering. The right figure displays the number of best score updates achieved by the acquisition function and recentering. From both figures, the acquisition function updates the best score 5 times within approximately 150 oracle calls, whereas the recentering fails to update the best score with about 350 oracle calls. These results demonstrate that the recentering wastes a huge amount of oracle calls.

Thank you for your response.

[W2, Q1] Evidence and analysis of misalignment problem

In Figure 12, how does it achieve zero dissimilarity? It seems strange. Is there any test data leakage?

Figures 7 and 8 show that the trained model of the proposed method only works for a single specific domain. Please see the concern below.

[W3, Q6] Diversity of experimental domains and general optimization ability of InvBO

It is the most serious concern. I think that the authors misunderstood my concern. The proposed algorithm is trained on each domain, which implies that the configuration used in this work cannot be used for unseen domains. Bayesian optimization is black-box optimization, so that an optimization method can solve any problem with a small amount of inductive bias. The authors tackled known domains accessing a true function.

If the proposed method can be applied in unseen tasks without re-training and with the same configuration, I can say that the proposed method is not domain-specific.

[Q3] Clarification on the appropriateness of the term 'Inversion'

I don't think your answer resolves my concern. First off, "inversion" in GAN inversion is not matched to "inversion" in this work. While GAN models an implicit distribution, the proposed method is defined on an explicit representation. Moreover, Equation (4) does not align with Figure 2(b).

[Q4, Q7] Details of the dimensionality of latent space and the Variational Autoencoder (VAE) used in experiments

Did you try to adjust the dimensionality of latent space? How does it impact on performance?

Most of my concerns haven't been resolved. I believe that the current manuscript is not ready to be published in NeurIPS.