IMPLICIT VARIATIONAL REJECTION SAMPLING

Thank you for your valuable feedback. We have carefully considered your comments, and here are our responses to each of your points:

1. On the Overstatement of Contribution: We appreciate the reviewer pointing this out and agree that the application of rejection sampling to an unnormalized posterior is a standard textbook approach. In the revised version, we will add more references to classical textbooks to downplay this point. However, we still believe that the introduction of this method is important, as it provides an effective solution to the problem of evaluating implicit variational distributions. We will adjust the language in the revised version to better clarify this.

2. On the Neural Network and Computational Requirements: We have already mentioned the increased computational demand when setting $M$ through cross-validation in the limitations section. In the revised version, we will emphasize more clearly the issue of training an additional neural network, particularly the computational overhead involved. This will help readers gain a clearer understanding of this limitation.

3. On the Empirical Hand-Tuning of Cross-Validation: We will further clarify that cross-validation is an empirical method and that the selection of $M$ is an experimental adjustment. This approach is based on iterative refinement through experimentation. In the revised version, we will make this more explicit to ensure the reproducibility and clarity of the method.

4. On the Setting of $M$ and the Rejection Rate: On page 5, we have already demonstrated the properties of the resampling distribution $r_{\theta, \phi}(z|x)$ and shown that it provides a better approximation compared to the implicit proposal distribution. We also pointed out that there is a trade-off: if the rejection rate is too high, it can hinder the model’s ability to efficiently obtain samples. Therefore, an appropriate choice of $M$ is crucial to ensure that the rejection rate remains at a practically suitable value. Based on our experiments, a high rejection rate will primarily affect sampling efficiency and will not typically lead to worse performance compared to plain variational inference. In the revised version, we will make this clearer and emphasize that high rejection rates generally do not cause a catastrophic failure of the method, but rather affect its efficiency.

Thank you again for the thorough review of our work. We will incorporate these suggestions to further improve the clarity and quality of the paper.

Part 2/2

4. Rejection Sampling and Termination:

Thank you for your insightful comment regarding the potential for rejection sampling to get stuck in an infinite loop, particularly with poor initializations of $q$, or suboptimal choices of $M$ and $T$. We fully acknowledge that, in such cases, the rejection rate could be high initially, and the algorithm might struggle to make progress, especially with bad initializations.

To address this issue, we implement a practical engineering solution: we introduce a maximum iteration limit for the rejection sampling process to prevent it from getting stuck in an infinite loop. This ensures that the algorithm terminates after a reasonable number of attempts, even in cases of particularly challenging initializations.

Furthermore, we have emphasized in the limitations section of the paper that our method relies on properly chosen hyperparameters, such as $M$ and $T$. As you pointed out, these parameters play a key role in the efficiency and convergence of the sampling process. We encourage careful tuning of these parameters to avoid issues with slow convergence or excessive rejection rates.

We appreciate your time and insightful comments, which have helped to significantly improve the clarity and rigor of our manuscript.

Part 1/2

Response to Reviewer Comments

Thank you for your thoughtful feedback. Below, we address each point raised in your review.

1. Novelty of Rejection Sampling in Implicit Models:

We appreciate the reviewer’s recognition of the extension of rejection sampling to implicit models. Indeed, as pointed out, the idea of using rejection sampling to refine variational distributions has been explored in previous work, such as Grover et al. (2018), for explicit models. However, our work extends this idea to implicit models, which introduces additional challenges such as the need for an effective method to approximate the posterior distribution within a neural network framework. We argue that this novel extension in the context of implicit models brings significant value, and we have highlighted this distinction in the revised manuscript to ensure that this contribution is clearly stated.

2. Toy Evaluation and Figure 1:

Thank you for pointing out the concerns regarding the color gradient on the contours in Figure 1. We would like to clarify that the color gradient represents the density of the generated samples, with red indicating higher concentration and blue indicating sparser regions. While the toy dataset is relatively simple, numerical methods inherently introduce some level of numerical error. For rejection sampling, although we can approximate the true distribution, we still encounter unavoidable numerical errors, and as such, perfect recovery of the true distribution is not always feasible.

Regarding the use of Negative Log-Likelihood (NLL) as the evaluation metric, we chose this because the resampled distribution is difficult to express with an explicit mathematical formulation. NLL serves as a simpler and more direct measure, which is also more accessible for readers. In contrast, more complex metrics like Wasserstein distance and C2ST are more precise but are harder to understand and simulate, especially for non-expert readers. Thus, we opted for NLL as the evaluation criterion for ease of understanding.

3. VAE Evaluation and Table 3:

We would like to clarify our approach on the following points:

Use of Published Results and Hyperparameters:

First, we believe that using published results from previous works is a widely accepted practice in the academic community. While hyperparameter settings may vary across studies, we have provided citations for the reference works, allowing readers to check the specific hyperparameter configurations used in those studies. This ensures transparency and enables readers to understand the context of the comparisons we present.

Addressing the Concern About Older Baselines:

We acknowledge that the baseline VAE methods referenced in our work are relatively older, and we appreciate your constructive criticism in this regard. In response, we have updated the manuscript to include a discussion of the two more recent works you mentioned—[1] Samarth Sinha et al. (2022) and [2] Louay Hazami et al. (2022). These papers present advancements that achieve improved results on MNIST, and we have now incorporated these into the discussion of our approach.

Expansion to CIFAR-10 and the 10-Page Limit:

In response to your suggestion to evaluate on more challenging datasets, we have expanded the manuscript to its maximum page limit (10 pages) and added experiments on CIFAR-10. This additional evaluation provides a broader context for our method’s performance and strengthens the empirical evidence supporting our claims.

Clarification on MNIST Results and NLL Score:

We would also like to clarify our perspective on the NLL score of 81.78 for MNIST. Our focus in this work is on using rejection sampling to improve implicit variational inference (VI), and VAE serves as a concrete application of VI to demonstrate the advantages of our approach over existing implicit VI methods. We did not intend to outperform all current VAE methods but rather to show that our method improves implicit VI. As deep learning techniques, particularly deep architectures, have become increasingly effective at extracting image features, the two works you referenced benefit from deep architectures in their encoders, which differ from the mathematically-driven improvements we propose. However, we have updated the manuscript to highlight how our approach can complement these recent methods, combining the best of both worlds.

We believe these revisions address your concerns and further substantiate the contributions of our work. We hope the expanded experiments and updated discussions provide a more comprehensive understanding of the method's capabilities.

We thank the reviewer for their constructive feedback and thoughtful questions, which allow us to clarify key aspects of our work and its relationship to prior research. Below, we address the concerns and questions point by point.

---

Distinction from Related Works [1], [2], and [3]

1. Distinction from [1]:
   • As noted by the reviewer, [1] reformulates the prior as a resampled distribution, whereas our method explicitly derives a new evidence lower bound (IR-ELBO) based on rejection sampling. While both approaches use density ratio estimation, our work focuses on leveraging rejection sampling to construct a tighter variational objective by using implicit distributions to directly approximate the posterior. We will revise our submission to make this distinction clearer.

---

2. Distinction from Related Work [2]

• While Equation (2) in [2] and our Equation (13) share a similar mathematical form, a key distinction lies in the nature of the variational distribution $q_\phi$: in [2], $q_\phi$ is an explicit distribution, whereas in our work,$q_\phi$ is an implicit distribution. This distinction is crucial as it aligns with our goal of improving implicit variational inference by leveraging rejection sampling, enabling the construction of more flexible posterior approximations.

• Moreover, while we have acknowledged the contributions of [2] in the related work section and expressed gratitude for their pioneering efforts, we will further clarify in the revised manuscript how Equation (13) was designed specifically to enhance implicit variational inference, a direction distinct from the explicit approach in [2].

---

3. Clarification of IR-ELBO Definition:
   • We appreciate the feedback regarding the confusion in the description of the IR-ELBO. To clarify: Equation (16) is not the full IR-ELBO but a key term within it. The IR-ELBO is derived by substituting this term into Eq. (15). The notation in Algorithm 2 (row 4) should be adjusted for consistency, and we will revise the manuscript to eliminate this ambiguity.

---

Questions

1. Is IR-ELBO looser than Eq. (14)?

---

• The confusion likely arises from our application of the Jensen inequality in deriving Eq. (16). This step is a standard mathematical operation that exchanges the expectation and logarithm to ensure an unbiased estimate of the IR-ELBO, rather than an indication of a looser bound.

• Our primary objective with the IR-ELBO is to improve implicit variational inference by leveraging the rejection sampling mechanism, which allows us to refine the posterior approximation. This approach is designed to enhance flexibility and accuracy in implicit variational inference, addressing some limitations of traditional methods. We will revise the manuscript to make this mathematical treatment and motivation clearer to avoid potential misunderstandings.

---

2. Is Eq. (4) correct?

---

• Yes, Eq. (4) is correct. Similar to prior works on implicit distributions, our formulation describes the process where a standard normal sample is passed through a neural network to output an implicit distribution. While this notation may appear slightly unconventional at first glance, it is consistent with standard practice in implicit variational inference.

---

3. Could IWRS be used for mixtures of variational distributions?

   • Yes, IWRS could be adapted to mixtures of variational distributions. Leveraging mixtures could improve the expressivity of the approximate posterior and may align well with the strong results from [4] and [5]. This is an exciting direction for future work, and we appreciate the suggestion.

4. Could the importance-weighted ELBO [6] be applied to Eq. (16)?

   • Indeed, combining the importance-weighted ELBO (IW-ELBO) with Eq. (16) could yield a tighter bound. This approach would require integrating importance weighting within the rejection sampling framework, which could amplify the benefits of both techniques. We will consider this as a potential extension in future work and briefly discuss this possibility in the final paper.

---

Proposed Revisions

1. Expand Section 3 to explicitly distinguish our approach from [1], [2], and [3], with added citations for related equations.
2. Revise Algorithm 2 to correctly define IR-ELBO and ensure consistency with the main text.
3. Clarify the role and notation in Eq. (4) with additional explanations.
4. Add a discussion of the potential for combining IWRS with mixtures of variational distributions and the IW-ELBO to strengthen the posterior approximation.

We thank the reviewer again for their detailed and insightful comments. These clarifications and revisions will significantly improve the quality of our submission.

We appreciate your thorough review of our submission and your valuable feedback. Below, we address your questions and concerns in detail:

---

1. Robustness of the Method in High-Dimensional Settings

We acknowledge the concern regarding rejection sampling's challenges in high-dimensional scenarios due to the curse of dimensionality. To mitigate this, our approach leverages implicit proposal distributions parameterized by neural networks, which are designed to closely approximate the target distribution. This reduces the rejection rate and improves efficiency even in higher dimensions.

In our experiments, we have evaluated the proposed method on datasets with moderately high-dimensional posterior distributions (e.g., 100 dimensions). Results demonstrate that the acceptance rates remain manageable, and the method performs favorably compared to baseline variational inference (VI) approaches. We recognize, however, that testing on more extreme high-dimensional cases (e.g., thousands of dimensions) could provide additional insights. This is a valuable direction for future work, and we are actively considering implementing more scalable architectures to further explore this aspect.

---

2. Computational Efficiency Compared to Other Methods

We agree that computational overhead is a critical factor in assessing the practicality of our approach. The addition of the discriminator network introduces some computational cost, particularly for estimating the density ratio and acceptance probability. To address this concern:

• Comparison with Baselines: In our experiments, we observed that the computational cost of training the discriminator is offset by the reduction in bias due to tighter approximation of the Evidence Lower Bound (ELBO). Compared to standard VI methods, our approach exhibits a trade-off where the marginal improvement in accuracy justifies the additional computation.
• Efficiency in Large-Scale Settings: To improve scalability, we used a lightweight architecture for the discriminator and optimized its training through mini-batch techniques. However, we acknowledge that in large-scale datasets or extremely high-dimensional problems, further optimization (e.g., parallelization or approximate methods) may be necessary.

We will enhance the discussion in the paper to clarify these trade-offs and include a comparison of wall-clock runtimes with baseline methods to better illustrate the computational efficiency of our approach.

---

3. Manual Tuning of the Hyperparameter $M$

We acknowledge the concern about hyperparameter tuning. The parameter $M$, which controls the acceptance threshold, is indeed tuned via cross-validation. While this introduces additional complexity, we note that $M$ has an interpretable role in balancing accuracy and efficiency. In practice, we found that the method is robust to small changes in $M$, and default values often work well.

In future iterations, we aim to explore adaptive methods for dynamically setting $M$ based on the observed statistics of the proposal and target distributions. This would reduce reliance on manual tuning and enhance usability in large-scale settings.

---

We thank you again for your constructive comments, which have helped us identify key areas for improvement. We are confident that addressing these concerns will strengthen the contribution and impact of our work.