SENSITIVITY-INFORMED REGULARIZATION FOR OFFLINE BLACK-BOX OPTIMIZATION

5. Hyper-parameter Tuning for $\omega$. Indeed, setting such hyper-parameters is non-trivial. We have however detailed several strategies to do this via building pseudo-oracle based on generating synthetic data, such as those described above in response to Reviewer kNdc, which we quote below for your quick review.

To fine-tune the hyper-parameters of our regularizer, we can use the surrogate of the base method (which was trained on the offline data) as a pseudo-oracle, which helps evaluate design proposals generated by each specific configuration of the regularizer. For example, we can leverage the surrogate model of COMs as a pseudo-oracle to evaluate design proposals generated by the gradient ascent method (GA).

To demonstrate this, we conducted three experiments that mirrored those in our ablation studies in Section 5.3 with the key difference being the use of a pseudo-oracle in place of the true oracle. The results are promising, showing that this technique can successfully discover the same optimal hyperparameters as those determined using the true oracle.

This is visualized in the plots below,

(a) https://github.com/abcdefghhgfedcba/SIRO/blob/main/rebuttals/hyperparameters_selection/coms_oracle_change_alpha.png

which shows that the value of sensitivity threshold $\alpha = 0.1$ (GA-0.1) is the best, as previously demonstrated using true oracle in Figure 2b. Likewise, the plots at

(b) https://github.com/abcdefghhgfedcba/SIRO/blob/main/rebuttals/hyperparameters_selection/coms_oracle_change_n_gamma.png

and

(c) https://github.com/abcdefghhgfedcba/SIRO/blob/main/rebuttals/hyperparameters_selection/coms_oracle_change_omega_bound.png

also show that the best hyperparameters are:

No. of perturbation sample $m=100$ as the corresponding regularized baseline GA-100 is best among those in plot (b) and the bounds on mean and variance of the perturbation distribution are $([-10^{-3},10^{-3}],[10^{-5},10^{-2}])$ as the corresponding regularized baseline GA-0 is best among those in plot (c). These are the same tuning results found by the oracle in our ablation studies in Figure 2c and Figure 2d.

Please note that we only use the true oracle in the ablation studies (Section 5.3) which is necessary to show the isolated effect of each of the component. Our other experiments do not use the true oracle for hyper-parameter tuning.

Please also note that the indexing of baseline in plots (a) and (b) correspond to different indexing systems of the tuning parameter candidates.

6. What is the distribution over random $\mathbf{x}$ drawn from the input domain. To answer the reviewer’s question, the definition is generic but in our implementation, the expectation over the entire input space is approximated by the empirical expectation of the training input in the offline dataset. Alternatively, we can always find a low-dimensional embedding of the input (e.g., via a variational auto-encoder) and sample on that latent space, whose results can be decoded back to the original space.

7. Why is $\gamma_i$ drawn from a distribution with non-zero mean? In our work, we aim to find a model that is most robust against the most adverse (but small) perturbation. Thus, the mean and variance of the perturbation distribution is not set but rather learned adversarially (as described in the paragraph following Eq. (7)). This can also be seen from Algorithm 1, specifically in step 11. As a result of such adversarial training, the mean of the perturbation distribution appears to be non-zero. We describe the overall process in the last paragraph of Section 5.1 and the last paragraph of Section 5.3.

Overall, we believe we had addressed all your concerns. Please let us know if the response is satisfactory or if you still have follow-up questions for us. We will be happy to discuss further.

1. The rationale behind considering sensitivity across the entire input domain. To answer the reviewer’s question, the definition of sensitivity measure is generic but, in our implementation, the expectation over the entire input space is approximated by the empirical expectation of the training input in the offline dataset. Alternatively, we can always find a low-dimensional embedding of the input (e.g., via a variational auto-encoder) and sample on that latent space, whose results can be decoded back to the original space.

2. Will the approximation quality of $\Phi(\gamma_i; \mathbf{w})$ be affected at out-of-distribution regime. We would like to assert that the out-of-distribution phenomenon will not happen during the learning of $\Phi$. To see this, note that the erratic out-of-distribution prediction happens on the surrogate of the true oracle because it is trained on non-representative (offline) data observing only values of the oracle at the bottom 40-th percentile -- see COMS -- https://arxiv.org/pdf/2107.06882

On the other hand, the model $\Phi(\gamma_i,\mathbf{w})$ is used in the context of Eq. (10) which aims to approximate the level set in Eq. (8). This only involves the surrogate model from which we can sample representatively since (unlike the oracle) we have full access to its parameterization. Thus, the out-of-distribution will not happen. In addition, we note that $\gamma_i$ is a scalar, not a high-dimensional vector.

3. $\Phi(\gamma_i; \mathbf{w})$ is either $0$ or $1$. This is a misunderstanding. In our work, the output of $\Phi(\gamma_i; \mathbf{w})$ denote the probability that $\gamma_i$ belongs to the level set in Eq. (8). As such, its value is a continuous value between 0 and 1. It is therefore reasonable to approximate it with a neural network.

4. How tight is the bound of the sensitivity in Lemma 2? The tightness of the upper-bound of the sensitivity in Lemma 2 can be seen from the inequality $\mathbb{I}(x \geq \alpha) \leq \min(1, (x/\alpha)^n)$ -- see Eq. (22) & (23) in Appendix B which proves Lemma 2. This inequality is tight because when $x \geq \alpha$, both sides are 1. Otherwise, when $x < \alpha$, $(x/\alpha) < 1$ and so, $(x/\alpha)^n$ is very close to $0 = \mathbb{I}(x \geq \alpha)$ when n is large. Given this, the further use of first-order Taylor expansion should be acceptable as it is a common practice.

To evaluate the tightness of Lemma 2, we undertook a minor experiment to calculate the disparity between $S_\phi$ and $S_{\phi}^{+}$ , specifically $\|S_\phi − S_{\phi}^{+}\|$. The findings of this experiment are visualized in

https://github.com/abcdefghhgfedcba/SIRO/blob/main/rebuttals/tightness/tightness.png

where we present the average and standard deviation across batch data during 50 epochs of training the gradient ascent algorithm (GA) on two tasks: Superconductor and TF-BIND-8. The illustration reveals that the largest divergence $\|S_\phi − S_{\phi}^{+}\|$ is approximately 7.5%, which is sufficiently tight to affirm the accuracy of Lemma 2.

We will address the remaining questions in the next comment

1. the expectation over the entire input space is approximated by the empirical expectation of the training input in the offline dataset. The paper addresses the challenge of making reliable predictions beyond the scope of offline data. In response to question 2, the authors noted that the offline data is considered non-representative of the entire input space. Given this, how do we justify the rationale behind approximating the expectation over the entire input space using the empirical expectation derived from the training input in the offline dataset?

we can always find a low-dimensional embedding of the input (e.g., via a variational auto-encoder) and sample on that latent space, whose results can be decoded back to the original space. This might be questionable because of the non-representativeness of the offline data. If we can discover a lower-dimensional representation of the input that can be reliably reconstructed into the original space, I am wondering why we do not build a model for the function g using this reduced-dimensional representation and optimize the surrogate with it? It would significantly alleviate the challenge of offline black-box optimization.

2. I believe the key question here is whether $\gamma_i$ is a scalar or a high-dimensional vector. If it is a scalar, then I agree with the authors that the domain is sufficiently small to be accurately approximated with a neural network. However, the paper defines $\gamma_i = \omega_\mu + \omega_\sigma \epsilon_i$, where $\omega_\mu$ represents the mean of a multivariate Gaussian distribution with the same dimension as the neural network's parameters $\phi$ in Definition 1. Therefore, it would be beneficial to provide additional clarification on why it is a scalar value.

3. In the first line of Equation 9, the probability within the expectation should be an indicator function of values of either $0$ or $1$ (the only random variable is $\gamma$, so if $\gamma$ is given, $\gamma \in R_\alpha$ is either true or false). Please provide clarification on why it is a probability.

4. The author asserts that the bound is tight when $n$ is large. However, in the appendix following Equation (23), the author opts for $n=2$ which is not considered a large value.

I'm interested in understanding why we do not incorporate the first-order Taylor approximation from Eq. (14) into the definition of the sensitivity measure in Eq. (4) directly.

Sensitivity measure definition

The response appears to be inconsistent. The initial explanation (and my question) discusses the rationality of approximating the expectation over the input space with that over the offline data. However, in the latest response, the authors argue that the authors hypothesize a sensitive measure should be calculated on the offline data. Additionally, under this hypothesis, the question arises as to why the sensitive measure is not defined as an expectation over the offline data but rather as an expectation over the input space.

Learning of $\Phi(\gamma_i;w)$

The authors assert that they misunderstood my question regarding $\kappa_i$ when stating that $\gamma_i$ is a scalar value. This is perplexing, as the initial response from the authors focused entirely on $\gamma_i$, with no mention of $\kappa_i$.

I am also puzzled by the connection between the factorized Gaussian distribution of $\gamma$ and the learning of $\Phi$. In the response, the authors mentioned, "with $m=100$ samples of $\gamma$, we have a set of $m=100$ scalars per dimension to sufficiently represent its $1$-dim Gaussian distribution." Does this imply that if $\gamma$ is $500$-dim, only $500$ samples (dimension independent?) of $\gamma$ are needed to find a neural network that approximates $\Phi(\gamma_i;w)$? To approximate $\Phi(\gamma_i;w)$, I believe a substantial number of samples is necessary to adequately cover the distributional support of $\gamma$. This becomes particularly challenging for high-dimensional $\gamma$, even when considering the factorized distribution of $\gamma$. In fact, when the distribution of $\gamma$ is factorized, the support of the distribution becomes even broader compared to the scenario where it is highly correlated.

$n=2$

Although there might be additional intricacies in plotting the approximation of $S_\phi^+$, I believe that examining the function $(x/\alpha)^n$ can provide insights into its proximity to zero, particularly when comparing the case of $n = 2$ to a larger value of $n$. Therefore, it appears contradictory that the authors advocate for a large value of $n$ while simultaneously opting for $n = 2$.

First-order Taylor approximation in Eq. (4)

I do not understand the authors' statement that "we cannot directly apply the first-order Taylor approximation from Eq. (14) to Eq. (4)". I can just utilize the identical Taylor approximation from Equation (14) for Equation (4): $|\mathbb{E}[g(x;\phi + \gamma)] - \mathbb{E}[g(x;\phi)]|= |\triangledown h(\phi)^\top \gamma|$

Since, $\gamma$ follows a multivariate Gaussian distribution, $\triangledown h(\phi)^\top \gamma$ follows a Gaussian distribution. Then

\\text{Pr}\_{\\gamma}(|\\mathbb{E}[g(x;\\phi + \\gamma)] - \\mathbb{E}[g(x;\\phi)]| \\ge \\alpha) = \\text{Pr}\_{\\gamma}(|\\triangledown h(\\phi)^\\top \\gamma| \\ge \\alpha)

It is only a combination of two univariate Gaussian CDFs. Hence, this greatly simplifies the calculation process.

First-order Taylor approximation in Eq. (4)

Thank you for this interesting suggestion.

Now, we realize and agree that indeed the Taylor approximation can also be plugged into the above formulation of $S$, making it differentiable.

This is a feasible alternative to our approach and we will inspect it in our revision. However, we do not think this invalidates our approach which is based on an empirically tight bound. It also provides significant improvement across a diverse set of baseline as shown in Table 1.

Overall, it is an interesting exploration for future follow-up but a thorough investigation of this approach is currently out-of-scope for this paper.

1. Showing some surrogate prediction results with the regularization term. Our regularization technique is designed to develop a more resilient surrogate model, capable of handling minor disturbances and delivering accurate predictions in out-of-distribution regimes.

To assess the precision of predictions using SIRO versus the original baseline, we carried out a small-scale experiment. In this experiment, we calculated the root-mean-square-error (RMSE) of surrogate predictions for designs suggested by the gradient ascent method (GA), with the ground truth being the actual objective value determined by the oracle.

The outcomes, including the mean and standard deviation of RMSE, are displayed in

https://github.com/abcdefghhgfedcba/SIRO/blob/main/rebuttals/RMSE/RMSE.png

Throughout 50 steps of gradient ascent, the integration of SIRO resulted in the surrogate model making more accurate predictions than the original, particularly indicated by a lower RMSE and reduced standard deviation. This trend was consistent across both tasks, Superconductor and TF-BIND-8.

2. Variance of CMA-ES seems to increase with performance. We do not think this is a consistent pattern. In Table 1, this is only the case with Ant Morphology and D’Kitty. Note that for TF-BIND-10, the variance increases but the performance decreases. Overall, the variance of the regularized optimizer can increase (but not always) because it has more parameters than the non-regularized version.

3. REINFORCE with regularization always find the global optimum. Thank you for bringing our attention to this. Upon a closer look into this, we realize that the REINFORCE baseline is somehow sensitive to the randomization of the offline data. We have re-run it for 16 times with more randomization of the offline dataset, the reported performance is $0.964 \pm 0.096$. So, we do not think REINFORCE + SIRO baseline always finds the global optimum. It was a coincidence that REINFORCE + SIRO performs extremely well in one particular dataset generated by Design Bench. When we add more data variation to it, the performance is no longer perfect but it is nonetheless close to the optimal performance.

We hope the above have addressed all your concerns. Please let us know if you have any follow-up questions for us. We will be happy to discuss further.

Thank for your reply about the model prediction and experimental detail.

But I think the result does not convince me of the better prediction of the proposed method. Like I mentioned in the weakness part, the Superconductor benchmark does not have an exact function oracle, which may inherently have the sensitivity issue (seems not addressed in the previous rebuttal). On the TF-Bind-8 task with an exact oracle, the performance between SIRO and the original model seems at the same level to me.

Therefore, I think the additional result does not help me to better understanding the improved performance of SIRO.

We appreciate the Reviewer for the prompt feedback.

To facilitate a better understanding of SIRO's improved performance, we made an earnest effort to conduct additional experiments on the TF-BIND-8 and TF-BIND-10 tasks. Unfortunately, due to time constraints, we could only complete experiments using the CbAS baseline for TF-BIND-8 and TF-BIND-10. You can access the results at this link: https://github.com/abcdefghhgfedcba/SIRO/blob/main/rebuttals/RMSE/RMSE_CbAS.png

These findings illustrate that the utilization of SIRO aids surrogate models in generating more accurate predictions than the original baseline. We hope that these results adequately address the Reviewer's concerns.

Regarding the issues related to Superconductor task, we agree with the Reviewer's observation that the lack of an exact oracle function could potentially introduce sensitivity issues. In principle, we would prefer to avoid conducting experiments on this task. However, since most prior studies have carried out experiments on this task, we have included the results to provide readers with a comprehensive perspective. We concur with the Reviewer's concern that the Superconductor baseline may not be an ideal choice. However, it's important to note that this concern is rooted in how the benchmark itself was constructed, rather than any shortcomings in our approach.The issues were mentioned before but the benchmark was still published after that. Please refer to the link below for more details:

https://openreview.net/forum?id=cQzf26aA3vM&noteId=_Cs3nvyGs5J .

1. Including more experiments. Thank you for the suggestions. We report below the result of regularizing ROMA with SIRO:

As for other experiments regarding NEMO (Fu & Levine, 2021) and IOM (Qi et al., 2022), the authors did not release code so it is impossible to demonstrate the actual impact of SIRO on these optimizers (in their best implementation).

2. Compare with different regularization methods. We also provide extra experiments showing comparison between different regularization methods:

In particular, we have run additional experiments on the Superconductor and TF-BIND-8 tasks to compare the performance improvement achieved by regularizing the gradient ascent (GA) optimizer with our proposed SIRO regularizer and the traditional L1 and L2 regularizers.

The results are visualized in the below plot hosted on the same anonymized GitHub repository that contains our experimental code:

https://github.com/abcdefghhgfedcba/SIRO/blob/main/rebuttals/L1_L2/L1_L2.png

The results indicate that regularizing GA with L1 and L2 (GA w/ L1 and GA w/ L2) norms is not as effective as regularizing with SIRO (GA w/ SIRO). This is expected since SIRO is specifically designed to condition the output behavior of the model against adversarial perturbation while L1 and L2 only generically penalize models with high complexity, measured by the L1 and L2 norms of their parameters.

3. Including more baselines. While we agree with the reviewer that incorporating the suggested baselines are interesting to include and could strengthen our paper, we also believe that not including them in the current manuscript does not pose a weakness to our main claim, as elaborated below:

(1) We do NOT aim to develop a new regularized surrogate that introduces a new offline optimizer competing with existing ones. Instead, we want to develop an optimizer-agnostic regularizer (SIRO) that can be applied synergistically to a diverse set of existing optimizers, boosting their performance as shown in our experiments.

(2) Although the set of existing optimizers used in our experiment is not exhaustive, we also do NOT aim to build a regularizer that works with all optimizers, which might not be feasible. We only claim (in our 2nd contribution bullet) that our developed regularizer works well to boost the performance of a diverse (but not necessarily exhaustive) set of optimizers. Even so, this is still a solid and practical contribution as none of the existing optimizers outperform others in all tasks.

Overall, we agree having those experiments will strengthen the paper (and we have indeed generated new experiment results for some of the suggestions as presented above) but we hope the reviewer will consider our points (1) and (2) above and would not view these missing experiments as weaknesses to our contribution claim.

4. Typos in Eq. (6) and (7). Thank you for catching the typo. We have corrected them.

5. In Definition 1, does the input space mean the offline dataset? In Definition 1, $\mathfrak{X}$ is the entire input space that include the example inputs in the offline dataset. We will put a footnote in the paper to make this clear.

Thanks for your response. I still have the following concerns:

1. Regarding the NEMO code, although it is not open-source, the RoMA project has implemented NEMO. It would be interesting if the author could utilize this implementation available at (https://github.com/sihyun-yu/RoMA/blob/main/design_baselines/safeweight_latent/nets.py) to conduct experiments. Since both codes are based on the design-baselines framework, if the model is applicable, the author can directly make use of it.

2. In the third point raised by the author, there seems to be a potential misunderstanding of my intention. What I meant to convey was that if the author wishes to highlight the advantages of the regularization term, they can simply incorporate their own regularization term into existing model-based methods. This would effectively demonstrate the significance of their approach. For example, comparing it with COMs by conducting a comparison between Grad. Ascent+SIRO and Grad. Ascent+COMs, in order to investigate the effects of SIRO.

3. The feedback from Reviewer kNdc regarding the unsatisfactory performance of the 50th percentile and 75th percentile is intriguing. However, the current response lacks sufficient convincing discussion. Further discussion on this issue is requested.

1. Comparison with L1 and L2 regularization. As requested, we have run additional experiments on the Superconductor and TF-BIND-8 tasks to compare the performance improvement achieved by regularizing the gradient ascent (GA) optimizer with our proposed SIRO regularizer and the traditional L1 and L2 regularizers.

The results are visualized in the below plot hosted on the same anonymized GitHub repository that contains our experimental code: https://github.com/abcdefghhgfedcba/SIRO/blob/main/rebuttals/L1_L2/L1_L2.png

The results indicate that regularizing GA with L1 and L2 (GA w/ L1 and GA w/ L2) norms is not as effective as regularizing with SIRO (GA w/ SIRO). This is expected since SIRO is specifically designed to condition the output behavior of the model against adversarial perturbation while L1 and L2 only generically penalize models with high complexity, measured by the L1 and L2 norms of their parameters.

2. Considering similar idea of robust Bayesian optimization in [1]. Thank you for the insightful suggestions. We believe the idea presented in [1] can be repurposed to fit into our context but that will be non-trivial since [1] defines the adverse effect of perturbation in terms of the maximum value a perturbed model can achieve -- see Eq. (4) in [1] -- which does not necessarily measure how susceptible a model is to perturbation.

For example, a low-output model can have its output doubled at the maximum adverse perturbation, but the final output is still less than that of another high-output model whose output only changes by 1% at the worst perturbation. In this case, the proposed algorithm in [1] will favor the former while ours would favor the latter.

This conceptual discrepancy stems from a key difference in focus between our work and [1]. In particular, [1] is set in a context where the regularization is applied directly to the function that needs to be minimized, assuming that it is either known or can be queried. As such, it makes sense that the optimizing algorithm in [1] has an implicit bias that low-output models can tolerate more perturbation, which is true in its context. In contrast, our work is set in a more sophisticated context where the regularization is applied to a learning model of the (unknown) function being optimized. As such, unlike [1], we cannot prioritize surrogates with low prediction over surrogates with high prediction.

But nonetheless, we agree that the idea in [1] is very intriguing and we will explore potential adaptation of it into offline optimization in our follow-up work. We note that this is also similar in spirit to the central idea in RoMA [2], another baseline that we had compared against. Thank you again for this interesting suggestion.

[1] Bertsimas, D., Nohadani, O., & Teo, K. M. (2010). Robust optimization for unconstrained simulation-based problems. Operations research, 58(1), 161-178.

[2] Yu, Sihyun, Sungsoo Ahn, Le Song, and Jinwoo Shin. "Roma: Robust model adaptation for offline model-based optimization." Advances in Neural Information Processing Systems 34 (2021): 4619-4631.

3. Improvement at 50th and 75th percentile. We agree with the reviewer that results for the 50th and 75th percentile are less impressive than reported results for the 100th percentile. This is potentially because most base methods are also not very effective at those percentiles. It is observed in both Tables 2 and 3 that the 50th and 75th solutions of the base methods often do not improve much over the empirical best. Thank you for pointing this out. This could be one potential venue of focus for our future expansion of the current idea.

We will address your remaining questions in the next comment.

4. Tuning hyper-parameters of the proposed regularizer. To fine-tune the hyper-parameters of our regularizer, we can use the surrogate of the base method (which was trained on the offline data) as a pseudo-oracle, which helps evaluate design proposals generated by each specific configuration of the regularizer. For example, we can leverage the surrogate model of COMs as a pseudo-oracle to evaluate design proposals generated by the gradient ascent method (GA).

To demonstrate this, we conducted three experiments that mirrored those in our ablation studies in Section 5.3 with the key difference being the use of a pseudo-oracle in place of the true oracle. The results are promising, showing that this technique can successfully discover the same optimal hyperparameters as those determined using the true oracle.

This is visualized in the plots below,

(a) https://github.com/abcdefghhgfedcba/SIRO/blob/main/rebuttals/hyperparameters_selection/coms_oracle_change_alpha.png

which shows that the value of sensitivity threshold $\alpha = 0.1$ (GA-0.1) is the best, as previously demonstrated using true oracle in Figure 2b. Likewise, the plots at

(b) https://github.com/abcdefghhgfedcba/SIRO/blob/main/rebuttals/hyperparameters_selection/coms_oracle_change_n_gamma.png

and

(c) https://github.com/abcdefghhgfedcba/SIRO/blob/main/rebuttals/hyperparameters_selection/coms_oracle_change_omega_bound.png

also show that the best hyperparameters are:

No. of perturbation sample $m=100$ as the corresponding regularized baseline GA-100 is best among those in plot (b) and the bounds on mean and variance of the perturbation distribution are $([-10^{-3},10^{-3}],[10^{-5},10^{-2}])$ as the corresponding regularized baseline GA-0 is best among those in plot (c). These are the same tuning results found by the oracle in our ablation studies in Figure 2c and Figure 2d.

Please note that we only use the true oracle in the ablation studies (Section 5.3) which is necessary to show the isolated effect of each of the component. Our other experiments do not use the true oracle for hyper-parameter tuning.

Please also note that the indexing of baseline in plots (a) and (b) correspond to different indexing systems of the tuning parameter candidates.

5. Does regularizing for smoothness make the trained parametric surrogates behave more similarly to Gaussian processes with standard kernels? Although the Gaussian process (GP) is a prior over a space of smooth functions, its prediction is sensitive to the kernel’s length-scales (one per input component) which are learned based on the offline data, and might not match the actual length-scale in an out-of-distribution regime. This is especially the case in non-static environments where the length-scales tend to vary across the input space. As such, a non-regularized GP will not necessarily behave similarly to a regularized parametric model using our notion of sensitivity in Definition 1 which applies to the entire output space. That being said, we want to emphasize that our proposed regularizer is model-agnostic and can be applied to regularize a GP-based surrogate too. Although it is beside the main point of this paper, it would indeed be interesting to inspect how a regularized GP surrogate performs in an offline optimization context. We will explore this in our follow-up work.

Thank you again for your suggestion. We really appreciate your insightful comments. We are looking forward to hearing back from you soon. Please also let us know if you have any follow-up questions for us. We will be very happy to discuss further.

Thank you for the detailed response. I appreciate the thoughts re. my comments that may have been slightly out of the scope of the original contribution. Though I still have some concerns about the general applicability of the method (e.g., the authors acknowledge performance could be improved at the 50/75 percentiles), I believe the added results improve the contribution of the work. My favorable impression of this paper remains, and I have increased my 'Contribution' score accordingly.

Thank you for the message. We understand and would like to continue our academic discussion.

Learning of Phi:

We appeared to infer that you were concerned about the high-dimensionality issue of learning $\Phi$, which can involve both input and output. Hence, we wanted to make it additionally clear that the output is a scalar. We had this impression because in the original message, your next question to the inquiry on $\Phi$ is about its output. Again, we apologize for the confusion.

Learning using $m$ samples of $\gamma$:

We wrote in our previous message to you that

Next, to the main discussion point, we believe the concern is that $\Phi$ might not be learned properly if a finite set of $m$ samples of $\gamma$ is not sufficient to represent its space

Hence, this will be resolved if we can show that $m$ samples of $\gamma$ are sufficient to compute its distribution parameter which succinctly represents its space

As we had proved, with $m$ samples of $\gamma$, the MLE model can predict the true mean of $\gamma$ distribution with a small variance. Thus, though the MLE is not the main learning task here, it serves to alleviate the concern that we would need a lot of samples to learn the true space of $\gamma$. Intuitively speaking, if the $m = 100$ samples of $\gamma$ contain enough information about its space and generation distribution, learning $\Phi$ would not be affected by the high-dimensional nature of $\gamma$.

Large value of $n$:

We appreciate your suggestion that increasing the value of $n$ would make the bound tighter. We acknowledged that.

But again, our point is still that with $n = 2$, we achieved good results and the bound looks tight empirically. From that point of view, we do not think it is necessary to tighten the bound further.

In addition, we now understand better what intricacies you were suggesting. But, those intricacies remain with larger values of $n$ so we are skeptical that we can get rid of those by plotting with increasing $n$. Alternatively, if your suggestion is more about a theoretical analysis of $(x/a)^n$ in this context, it is highly non-trivial and is perhaps beyond the current scope. Do you have a suggestion for us on this? We appreciate and highly value your technical suggestions.

Overall, we agree with the reviewer that there is value in investigating the performance of an implementation with higher values of $n$ but this is more about an imperial implementation rather than a main contribution. In addition, we seek the reviewer’s understanding that our compute resource is tight during the rebuttal period so it is hard to repeat the entire set of experiments with larger values for $n$.

Taylor Approximation in Eq. (4)

We have acknowledged that it is a good direction and we will explore it in a future follow-up but for now, it is an alternative solution to the same problem (i.e., using Taylor approximation to reduce the computational cost) and as such, we believe it is orthogonal to our contribution, e.g. even if it does improve the performance further, such improvement does not negate the performance that we have achieved and reported here.

We do agree that your suggestion has the possibility of removing stringent approximations. However, it will require substantial investigation and thus, this idea deserves a separate treatment in another paper.

Overall, thank you again for keeping the discussion alive. Your suggestions are potential directions to improve our current paper, which we will investigate in a future follow-up.

I apologize the authors for the statement about insincerity. This perspective arises from my personal interpretation, and I acknowledge that the authors may have rightfully pointed out that we should not discuss or imply this aspect in academic discussions. I learned a valuable lesson from this experience.

Regarding the experiment plot, my intention was to highlight that analyzing the function $(x/\alpha)^n$ provides a simpler approach. On the other hand, creating a plot for the approximation of $\mathcal{S}_{\phi}^+$ introduces numerous intricacies, including randomness, sample size variations in $\gamma$ and the distribution of $\gamma$. I apologize for any confusion caused by my lack of clarity.

Although there might be additional intricacies in plotting the approximation of $\mathcal{S}_{\phi}^+$ , I believe that examining the function $(x/\alpha)^n$ can provide insights into its proximity to zero, particularly when comparing the case of $n = 2$ to a larger value of $n$.

Nevertheless, my perspective on the paper remains, as I highlighted several major problems in my last response to the authors that have yet to be resolved. This is different from the authors' perspective in the summary.