Incorporating Domain Knowledge in VAE Learning via Exponential Dissimilarity-Dispersion Family

Thank you for reviewing the manuscript. We will answer your questions below.

The general derivation is quite similar to $\sigma$-VAE, except for the extension of MSE to different dissimilarity functions. The σ-VAE proposes an approach to resolve the balancing and tuning issues faced by β-VAE and indeed shares a similar concept with our paper. While the σ-VAE focuses on calibrating β in the decoder, our proposed method significantly differs by using dissimilarity functions to extend the functionality of VAEs themselves, expanding their theoretical framework while maintaining probabilistic consistency. Additionally, our method allows for the automatic adjustment of scale parameters, eliminating the need for manual tuning of the decoder's variance, unlike σ-VAE. Moreover, experiments have confirmed that our proposed method demonstrates equal or superior performance to σ-VAE. We believe these novelties represent a significant advancement in the extensibility of VAEs beyond what σ-VAE offers.

When choosing MSE as the similarity metric, why does there exist a performance difference between $\sigma$-VAE and the proposed method in Tab.3? This aspect is one of the key points where the advantages of our proposed method are most evident. As stated in Viewpoint (i) of the introduction, the focus of our paper is the necessity of introducing a family of distributions to utilize prior information in VAEs, which already have a theoretical foundation. While previous studies have attempted to introduce additional mechanisms to balance reconstruction and regularization errors (e.g., $\sigma$-VAE, Rybkin et al., 2021), these approaches often deviated from the probabilistic framework, posing a challenge. Thus, our paper aims to resolve these traditional issues by attempting to learn using domain knowledge in the form of dissimilarity functions while maintaining the probabilistic model framework inherent in VAEs. Specifically, we propose a unified approach that balances reconstruction and regularization through variational inference backed by a probabilistic understanding of distribution families, rather than resolving issues through additional mechanisms or regularization. The experimental results in Table 3 show that the advantages of $\sigma$-VAE can be replicated in comparison with β-VAE, and these benefits are demonstrated. Furthermore, our proposed method has shown superiority over these techniques, particularly demonstrating the effectiveness of automatic tuning including the dispersion $\gamma$ in complex datasets such as Food-101 and CIFAR-10. This indicates that the proposed method, compared to $\sigma$-VAE, provides a more nuanced capture of data characteristics, which can lead to differences in performance.

In Tab.5, different dissimilarities ... tuned on the validation set? First, regarding Table 5, the hyperparameters for both the baseline and proposed methods were determined based on the FID from the validation set. The dissimilarity functions selected in our method are detailed in Figure 4. As for Table 3, rather than optimizing β or evaluating the highest quality resolution, we focused on validating how different values of β affect performance. This allows us to directly observe the effects of changes in β. We compare these performances to those of the automatically determined γ. We will add to the revised manuscript the points that were insufficiently explained.

If the explanations provided have resolved your concerns, we would appreciate it if you could reconsider the score given.

We appreciate your inquiry and thank you for your advice. We would like to make sure our response meets your needs. If you have any more questions or if there's anything in our response that doesn't fully address your concerns, please let us know. Your feedback is important to us and we'd be happy to discuss further. With just two days left in the discussion period, we're wondering if you've had a chance to review our response and if there's anything else we can help with.

Thanks for your reply! My concern remains.

• When using MSE as the similarity metric, mathematically, I do not see any evident difference between the proposed method and $\sigma$-VAE (Rybkin et al., 2021). In Rybkin et al., 2021, the authors have mentioned just below eq(6) that $\sigma$ can be selected by minimizing the weighted MSE loss, which is indicated as the optimal $\sigma$ and also does not require the manual tuning of $\sigma$.

• W.r.t the experiments:

  • In Table 3, for SVHN, $\beta=0.1$, you have FID = 15.97, KID=0.0897, the FID is very close to that of the proposed method in Table 5, and the KID is much better. Moreover, the results of $\beta$-VAE in Table 3 are only based on six choices of $\beta$. My point is: why not compare to $\beta$-VAE with more choices of $\beta$ in Table 5 to illustrate that the proposed method really outperforms?
  • How are the hyper-parameters tuned for the baselines? I want to see how you set the hyper-parameter range for each baseline and what are the specific hyper-parameters for each baseline. I have already noted what you marked as blue when I was reviewing the paper.
  • As you have incorporated the "domain knowledge" (actually weird to say so, because it's not predefined, but tuned as a hyper-parameter) in the form of dissimilarity functions to achieve better performance, why not directly indicate in Table 5 the chosen dissimilarity function? (I guess you mean detailed in Table 4?) I can relate this information through the two tables, but it takes additional time for the readers.

Thank you for reviewing the manuscript. We will answer your questions below.

Weaknesses: The paper's contribution is limited, ... around this idea. The novelty of our proposed method lies not just in suggesting a new dissimilarity function but in providing a unified framework for employing such functions within VAEs across various fields, along with a methodology for the automatic estimation of scale parameters. We have established a new theory that incorporates domain knowledge into VAEs in a probabilistically consistent manner, demonstrating the potential to integrate insights from various domains into VAEs as functional biases in a unified way. As VAEs are one of the representative generative models alongside GANs and diffusion models and are used in various fields and practical challenges, the application range of our proposed theory is expected to expand broadly. This paper constructs and validates the foundation of this new theory. Although the current contribution of this paper has not yet extended to direct applications in GANs or diffusion models, we believe it provides valuable insights for the deep learning and generative modeling communities.

Questions: How do you extend this idea to other generative models such as GANs or latent diffusion models? VAEs, GANs, and diffusion models each offer distinct advantages as generative models. In this paper, we propose enhancements to the versatility of VAEs, which have a probabilistic foundation. Meanwhile, GANs and diffusion models are garnering attention for high-quality image generation based on large datasets and substantial model sizes. It is conceivable that considering data characteristics and domain knowledge will become important in these models as well, where this paper can make a contribution. Specifically, the stochastic processes inherent in diffusion models may offer a higher potential for applying the concepts of this theory. For instance, incorporating the concepts of this paper into the denoising process could be envisaged. We have added 'potential' as future work in the manuscript.

If the explanations provided have resolved your concerns, we would appreciate it if you could reconsider the score given.

We appreciate your inquiry and thank you for your advice. We would like to make sure our response meets your needs. If you have any more questions or if there's anything in our response that doesn't fully address your concerns, please let us know. Your feedback is important to us and we'd be happy to discuss further. With just two days left in the discussion period, we're wondering if you've had a chance to review our response and if there's anything else we can help with.

Thank you very much for your careful review of the manuscript, including checking the source code. We also thank you for the many suggestions for improvement of the manuscript. The following are the answers to the questions we received.

One aspect I am unclear about is why the reconstruction loss is not just defined directly .. (using Jensen's). In this paper, reconstruction loss was defined using a specific discrepancy measure d in order to capture the rich characteristics of the data in more detail. This measure captures the complexity of the data and allows for flexibility in the model. Using Jensen's inequality allows for more robust constraints to be applied to the loss function during the optimization process, resulting in more precise management of information transfer from latent variables to data variables.

The above perspective also skips the need for approximating ... took such a simplistic approach). Your point about not needing to approximate the normalization term in EDDF is correct. This approach was chosen to reduce computational costs and allow for practical applications. Indeed, for deterministic decoding, explicit densities are not required, but this may limit the versatility of the model. Therefore, we defined explicit densities to provide a more general framework applicable to different types of data.

One further note, the above perspective seems to ... general dissimilarity function). The "generalized variational inference" perspective you have introduced is perfectly in line with our research motivation. From this perspective, our approach extends the framework of probabilistic variational inference to handle a wider range of discrepancy functions. This allows for better adaptability to specific data sets and tasks. Thank you for your deep insight.

Eq. (20) is misleading as from Eq. (29) the equality should instead be an upper bound due to Jensen's. Thank you for pointing this out. We have clarified the text to avoid misunderstandings.

Why is the M / 2 factor not utilized in Eq. (24)? This was a typographical error and has been corrected. The implementation has been reconfirmed as correct.

Above Eq. (25) there is a mention of "well-trained autoencoder" .. instead? The wording has been revised and clarified.

It is worth clarifying what the regularization loss is in Eq. (25). ... the current text. The wording has been revised and clarified.

Something to add to Section 3.1: ... Section 2.7. Thank you very much for your helpful comments. We have added them to the manuscript.

Please note that some of the comments we have received are beyond our knowledge. We would appreciate your advice if the corrections are not appropriate. If the above has answered your questions, we would be happy to consider reviewing the score.

what’s the computational cost of the proposed method? As originally described in the architecture details in the appendix, the computation cost is about 1h using a single GPU (RTX 2080Ti). Other parameters are also originally described in the appendix.

Can you give some insights or explanations on why the proposed model performs better on some of the datasets but worse on others in Table 3? Using the visual domain as an example, this experiment comprehensively examines the dissimilarity functions proposed in the past, not limited to Neural Network-friendly similarity measures. Since the proposed method includes the aspect of gradient descent applicability as an implicit requirement, the applicability of the dissimilarity function may be quantified in terms of differentiability in the domain domain. Therefore, one drawback of the proposed method is that its performance is highly dependent on the Neural Network-Friendly dissimilarity function. The indicators of unstable performance are mainly those proposed before the rise of Deep Learning, and are considered to include operations that are not stochastic gradient descent-friendly. In such cases, it is considered necessary to introduce a more stable method of gradient calculation, as it may affect the stable learning of the proposed method. However, please note that neural network-based LPIPS and other methods are differentiable and thus provide stable learning. The above suggests that a differentiable data structure is required for the data domain, which is a limitation in the application of the proposed method.

Can you give any justifications on how the proposed distribution family encodes domain knowledge? In this paper, we have demonstrated the effectiveness of the theory primarily through quantitative considerations. We believe that qualitative evaluation is a better way to understand the effectiveness of encoding domain knowledge. Figures 3 and 4 in the Appendix show the qualitative evaluation results corresponding to Tables 3 and 5. It can be seen that the proposed method is able to reflect the style and details of the input images better than other methods on relatively complex datasets such as Food-101 and CelebA. This indicates that the proposed method encodes knowledge important to the domain.

If the explanations provided have resolved your concerns, we would appreciate it if you could reconsider the score given.

Thank you for reviewing the manuscript. We will answer your questions below. Please note that the 1-D validation is preliminary, and the main verifications in this paper are conducted using 2-D data.

Technical novelty: the section 4 on VAE optimization ... from existing methods (e.g., Rybkin et al., 2021; Lin et al., 2019)? The novelty of our proposed method lies in extending the applicability of VAEs, widely used across various fields, by proposing an approach that maintains the validity of the probabilistic model. Specifically, as mentioned in Viewpoint (i) of the introduction, the introduction of a family of distributions is necessary to utilize prior information within VAEs that are already theoretically substantiated. Previous research has attempted to introduce additional mechanisms to balance reconstruction and regularization errors (e.g., Rybkin et al., 2021; Lin et al., 2019). However, these approaches did not consider the probabilistic framework inherent to the original VAE, making probabilistic interpretation challenging even when accuracy was improved. Therefore, this paper aims to resolve these traditional challenges by attempting to learn using domain knowledge in the form of dissimilarity functions while maintaining the probabilistic framework of VAEs. We propose a unified approach to balance reconstruction and regularization through variational inference with a probabilistic foundation provided by distribution families. Our experiments have verified that this approach performs as well or better than conventional ones. By establishing and demonstrating a theory that incorporates domain knowledge into VAEs through a probabilistically consistent approach, we have shown the potential to integrate insights from various fields into VAEs as inductive biases in a unified way. Given that VAEs are already used in various fields and practical challenges, we believe the application range of our proposed theory will broaden, and this paper constructs and validates the foundational theory for it. At this stage, the contribution of our theory remains within the domain of VAEs, but we believe it provides valuable information for the community in the field of generative modeling.

Is there any theoretical guarantee or analysis ... to valid the claim. The main manuscript carries out validations on the quantification of approximations in a simple 1-D setting. Additionally, validations regarding the validity of γ<<1 are performed in appendix G of the original manuscript.

Significance: the significance of the paper is unclear to me. ... understand its significance. Indeed, the experimental results in Table 3 do not always show our proposed method as superior. However, the technical contribution of this paper is not to achieve the highest precision in a specific task but to derive a theoretical framework for the introduction of any dissimilarity into VAEs. The focus of the experiments in Table 3 is to verify the validity of the newly proposed EDDF family of distributions and their estimation algorithm within the unified framework across various datasets. The fact that our proposed method, which features automatic adjustment of the scale parameter, exhibits performance comparable to the existing literature confirms the effectiveness of our theory. Meanwhile, as pointed out, discussions on the relationship and applicability to other generative models are necessary. Thus, the revised manuscript includes additions to this aspect.

Presentation: The term ‘domain knowledge’ used ... well defined at all. In this paper, data characteristics are interpreted and discussed as distribution. An example of the relationship between distribution and EDDF decoder is shown in Table 1. The novelty of this paper is that it is possible to incorporate various data distribution assumptions in a unified framework based on probability theory. As you mentioned, the term "domain knowledge" is abstract and lacks direct explanation of the correspondence with the data distribution, so we have revised the manuscript to clarify.

We appreciate your inquiry and thank you for your advice. We would like to make sure our response meets your needs. If you have any more questions or if there's anything in our response that doesn't fully address your concerns, please let us know. Your feedback is important to us and we'd be happy to discuss further. With just two days left in the discussion period, we're wondering if you've had a chance to review our response and if there's anything else we can help with.

We apologize for a word error in our manuscript that appears to have led to a misunderstanding. In the explanation of the general structure of VAEs, the term "maximization" under Equation (1) should have been "minimization". As you correctly pointed out, the negative ELBO is indeed the subject to be minimized, and our treatment of ELBO optimization in VAEs in this paper is consistent with standard practice. This section pertains to the general formalization of VAEs and is not directly related to our proposed method. Given that the presentation scores from the other four reviewers were all highest, we believe that this single word error has not significantly impacted the overall manuscript. Moreover, the novelty of our proposed method is described in Sec. 3.

As we mentioned in the introduction, our novelty lies not in the minimization of ELBO but in proposing a framework that maintains the probabilistic model structure while incorporating domain knowledge into the optimization of ELBO, and we kindly request a review focused on the novel aspects. Thank you for pointing out this important issue. We are grateful for your meticulous and constructive review.

page 1 : "VAEs with the commonly used settings ... appear as undesired/bad qualities. Thank you for your comment. It has been corrected.

pages 2-3: "...with negative ELBO objective... objective is minimized. page 4, last equation ... eqs (9), (18). The error regarding minimization has been rectified.

Some suggestions: Thank you for your feedback. Indeed, we agree that conducting experiments with different random seeds is important for enhancing the reliability of the paper. As for the validity of the evaluation metrics, as stated above, we have utilized metrics suited for verifying our theory and selected for evaluating various dissimilarity functions, which we believe are appropriate for testing the effectiveness of the theory. Regarding the datasets, the selection is crucial for demonstrating our theory; thus, we have already performed validations with five different datasets, which is more extensive than in previous literature. Consequently, while we concur with the importance of experiments to assess the impact of random seeds in the camera-ready version, we cannot agree that it directly leads to a lower evaluation without considering other validations.

If the explanations provided have resolved your concerns, we would appreciate it if you could reconsider the score given.

The writing of the paper is somewhat obscure, especially in what concerns long sentences, cf below. Thank you for your feedback. We have revisited the entire manuscript and clarified the sentence accordingly.

The rational to consider the minimization, instead of the standard maximization, of the ELBO objective is not clear. This was a word error in the explanation of the general introduction of VAEs. We apologize for any confusion caused.

The results seems to be too incremental, many previous works have proposed to replace the Gaussian distribution in VAE by some other distributions. In this paper, we present not merely a substitution of simple distributions but a unified framework for dissimilarity functions in VAEs, along with methodologies to realize it while maintaining the validity of the probabilistic model. The differences between prior work and our proposed approach are primarily detailed in the related work section of the original manuscript; please refer to it. For instance, Czolbe et al., "A Loss Function for Generative Neural Networks Based on Watson’s Perceptual Model," 2020 and Esser et al., "Taming Transformers for High-Resolution Image Synthesis," 2021 employ perceptual loss. However, these maintain a constant scale in the distribution, which fails to leverage the Bayesian properties of VAEs, leading to a balancing problem where the decoder cannot capture uncertainty. Moreover, these perceptual losses are introduced in an ad-hoc manner, making it challenging to tie them to the rich theoretical foundation that VAEs possess. Our proposed method suggests a versatile dissimilarity function that preserves the theoretical coherence inherent in VAEs, allowing for the introduction of any function deemed significant for the data domain under consideration, as discussed in viewpoint II. Also, while Takida et al., "Preventing oversmoothing in VAE via generalized variance parameterization," 2022 is mentioned as prior work, their consideration is limited to Gaussian cases, making it difficult to incorporate data domain-specific dissimilarity functions.

In 2 out of 5 cases in the described empirical results, the baselines from previous papers performed better (Table 5) Indeed, the results in Table 5 do not always favor our proposed method. However, the technical contribution of this paper is not about achieving the highest precision in image generation tasks, but about the theoretical derivation and validation of a unified framework for introducing arbitrary dissimilarity into VAEs. Therefore, the most notable aspect of our experiments is the verification of whether the EDDF family of distributions and its estimation algorithm operate validly. The fact that our proposed method performs comparably to existing literature validates the effectiveness of our theory.

The standard deviations in the experiments are not reported. Only two out of many metrics evaluating VAE quality are calculated. In particular, the reconstruction errors are not reported in the comparison tables. The primary focus of our experiments is to demonstrate a theory that incorporates domain knowledge into the probabilistic model characteristics while preserving its properties. To validate this theory, it is necessary to comprehensively evaluate dissimilarity functions proposed in the past. Using these dissimilarity metrics directly as evaluation criteria would lead to an unfair assessment and could shift the focus away from the main objective of the verification, potentially causing confusion. Therefore, we uniformly employed the current metrics for this purpose. Consequently, we believe that FID and KID are suitable unified metrics for the evaluations in this verification and have adopted them. Furthermore, we have validated the effectiveness of our approach across a broader range of datasets compared to previous literature. We will address the standard deviation below.

Rather simple datasets are considered only. Also only the visual domain is considered. The primary application area for VAEs is the image domain, which also readily lends itself to the expression of domain knowledge, hence we have conducted validations using major image datasets. From the perspective of incorporating domain knowledge, considering the characteristics of the data is crucial. We believe that evaluations with five types of datasets are extensive compared to related prior research.

We appreciate your inquiry and thank you for your advice. We would like to make sure our response meets your needs. If you have any more questions or if there's anything in our response that doesn't fully address your concerns, please let us know. Your feedback is important to us and we'd be happy to discuss further. With just two days left in the discussion period, we're wondering if you've had a chance to review our response and if there's anything else we can help with.