CFASL: Composite Factor-Aligned Symmetry Learning for Disentanglement in Variational Autoencoder

1. The proposed approach incorporates a diverse array of loss terms within its training objectives, with each term potentially making a distinct contribution. However, this diversity comes at the expense of imposing significant assumptions on the underlying image distribution. While I acknowledge that these assumptions may be justified within the context of the datasets considered in this paper, it's worth noting that some metrics, such as DCI, do not unequivocally demonstrate superiority in the ablation study presented in Table 2.

Nevertheless, I believe that the paper could benefit from a more comprehensive exploration of the limitations stemming from these strong assumptions. It would be valuable for the authors to provide concrete examples where these assumptions result in unintended or adverse outcomes. Even for an unsupervised setting, it remains crucial to take into account the nature of transformations within the image domain. A more explicit discussion of these assumption-related limitations would substantially bolster the significance of the claims advanced in this paper, in my view.

2. The qualitative results exhibit low image quality. While this is common across unsupervised disentanglement methods, it is really challenging to get convinced that better disentanglement is achieved. It would be valuable for the author to consider domain-specific metrics for the evaluation phase e.g. face identity loss, facial expression classification, head pose regression, etc. to assess whether only a specific attribute is altered during the single factor change experiments.

Weaknesses summarized:

• The paper is challenging to read as the english is quite poor and the logical flow of the work is unorganized.
• The method itself is composed of a wide variety of loss terms and the intuition or reasoning for why these terms are necessary is not provided. (Specifically for the parallel and perpendicular losses).

In more detail:

Weakness 1: There are many typos and poor grammar throughout the paper, with many sentences simply not making much sense. I include a few examples below, but there are many many more and the authors should have someone proof read this work more carefully:

• In the abstract: "We propose ... (CFASL) on VAEs for the extension to [a] general multi-factor change condition without constraint."
• "To implement group equivariant VAE, Winter et al. (2022); Nasiri & Bepler (2022) achieve the translation and rotation equivariant VAE"
• "For the equivariant encoder and decoder, we differently propose the single forward process by the encoder and decoder objective functions compared to previous work (Yang et al., 2022)."
• "Differently, we induce disentanglement learning with group equivariant VAE for inductive bias."
• 'The unsupervised learning work (Winter et al., 2022) achieves class invariant and group equivariant function in less constraint condition.'

Weakness 2: Naming is extremely unclear. For example, what are 'sections' referred to in Section 3.2? How do these differ from factors?

Weakness 3: Despite appealing to a precise probabilistic generative model as its primary value and distinction from prior work, the model itself could be made significantly more elegant in the context of generative models. For example, the 'factor prediction' mechanism could be integrated as a component of the generative model and inferred with another approximate posterior, as done in prior work (Song et al 2023).

Weakness 4: The discussion of learning the Lie algebra is quite rushed and the intuition for why the large set of different loss terms should be incorporated is largely missing.

[1] (Song et al. 2023) https://arxiv.org/pdf/2309.13167.pdf

In this work, one of the important missing part is the proper probabilistic derivation of the methodology, the core of the VAE framework. Or it may be due to the way the paper/work has been presented. To me, it's not sufficient to connect to the VAE world. It is suggested the authors clarify this important aspect with necessary derivations.

For certain items/results, the authors claim statistical significance performance (section 5.2, and appendix D); however, without sufficient details of their significance tests. It is suggested authors include details of these statistical tests.

As the authors have implemented the benchmark approaches (section 5) by themselves, we may require additional details for a fair companion of their results.

The paper/research may have some significance, and it would be beneficial if the source code could be released.

1. Only simple datasets are considered, the method is not tested on standard complex datasets like MPI 3D.

2. Reported improvements of CFASL in all measured metrics are essentially always situated within standard deviations of some other methods.

3. Reconstruction loss is not reported in 3 out of 4 datasets. Upon visual inspection of reported samples, the reconstruction quality is not satisfactory.

4. As reported on Figure 4, on 3DShapes dataset, there is no consistent improvement in FVM metric even at the expense of deteriorating reconstruction quality .

5. There is no theoretic justifications for introduction of so many, seven in total, additional loss terms.

6. Description of Lie group action is not clear, how the action by psi_i is defined? how the dimensions of Lie groups are chosen?

7. The described group action by matrix multiplications do not preserve the normal distribution, so the group equivariant term is not compatible with the standard KL term from beta-VAE loss.

8. There is no comparison with most recent disentanglement methods like DAVA, TCWAE.

9. Related work section does not mention many works from vast literature on disentanglement learning, eg Disentangling Adversarial Variational Autoencoder (ICLR 2023).