Autoencoders with Intrinsic Dimension Constraints for Learning Low Dimensional Image Representations

The main goal is to find better low-dimensional representations for images using AutoEncoders. Therefore, the method is righteously tested on the usefulness of low-dimensional latent representations for downstream tasks. However, the proposed new learning objective does not deal with the lower-dimensional representations explicitly. Indeed, both encoder and decoder weights are trained to satisfy the GID and LID constraints. As no theoretical argument is provided for why the proposed penalty terms should make the low-dimensional representations better, it is not clear to me that the proposed objective leads to the desired result. What if only the encoder part is trained on the new objective involving the GID and LID constraints while training the decoder part on the reconstruction loss? This would, in my opinion, compensate for the lack of theoretical evidence and align the algorithm with the main goal of the paper.

The method crucially depends on the intrinsic dimensionality estimator used to calculate the GID and LID. In the paper, the effective number of principal components is used. However, this number was derived in the context of neural activity with no theoretical guarantee. It is not clear why this estimator is suitable for the image domain. Estimating the ID on image manifolds is actually an active area of research, see e.g. [1]. Overall, I would have appreciated a greater discussion on the choice of the ID estimator. Especially, could there actually be a drawback when using a wrong ID estimator? Also, since the ablation studies show the importance of LID, I would have liked to see a greater discussion on this new, somewhat non-standard notion of local intrinsic dimensionality. In that regard, I think the paper would benefit from a simple toy example where a known manifold, e.g. a swiss-roll, is learned using a standard AE and AE-IDC. Like this, the effect of different ID-estimators on the latent representation can be visualized. If the AE-IDC is able to unroll the swiss-roll providing a more meaningful latent structure than a classical AE, this would strengthen the main message of the paper by confirming the overall intuition. Of course, in that case, only the GID constraint can be used. However, extensions of that toy example to image manifolds are also possible, e.g. by applying the method on image manifolds generated by a StyleGan, see [1].

The experimental evidence provided is not strong in my opinion. I don't see any difference between the quality of learned representations of MAE and MAE-IDC on ImageNet10 (Figure 4 left). Also, the gain in classification performance on ImageNet-1K is not significant (0.1%). The latter suggests that the claim of the authors, namely that "ImageNet10 provides a fast test on ImageNet without loss of generality" is wrong. Thus the remaining results, which do show an advantage of the proposed method, could be due to the low number of classes which weakens the overall message of the paper.

On the basis of this analysis, I am leaning-to-reject the current version of the paper. However, I am happy to change my assessment if the authors can address my raised concerns.

I have some doubts on the relevance of this work to the ICLR community. Autoencoders, although historically important, are less relevant to today's generative modeling community due to the rise of high-capacity models such as diffusion models. It is sometimes okay for a work to focus on autoencoders, e.g. [0], especially for new mathematically sophisticated techniques for which theoretical analysis that is not yet possible on other kinds of models. It would be more relevant if the authors considered diffusion models. However I acknowledge that perhaps this work is a step towards using the regularizer in more advanced models.

The relationship of the proposed ID measure to other ID measures in the literature is not explored. For instance, does the ID measure in this work correlate with the other measures discussed in Section 2.2 on toy-benchmarks (e.g. hyper-spheres/cubes) or real-images?

One nitpick: in Section 2.3 you write that [1] showed "the dataset with higher ID value is more vulnerable to attacks", but actually that work does not consider adversarial attacks at all.

I am also not very clear on why the proposed local ID measure is actually local. In Section 3.1, the authors write "[c]onsidering that feature maps from the same data point are highly correlated", but I don't see why this is the case.

Regarding the cost of the regularizer, I read asymptotic complexity analysis in Section 3.3, but I still have some doubts it is not more expensive. Wall-clock timings for the regularized and unregularized would be helpful.

Another nitpick: the term "loss landscape" usually refers to the function L(w) for different weights w (often projected into a low-dim space). The authors use that term to describe the curves in the Figure 3. These are usually just called "loss curves".

Plots in Figure 4 are hard to read.

[0] Topological Autoencoders - Moor et al. (ICML 2020) https://proceedings.mlr.press/v119/moor20a.html

[1]The Intrinsic Dimension of Images and Its Impact on Learning - Pope et al. (ICLR 2021) https://openreview.net/forum?id=XJk19XzGq2J

W1) Among the technical contributions of the paper, the only novel idea is a rather straightforward regularization based on differences of IDs (global and local) between the inputs and their reconstructions. The relationship mentioned in S1 between local distance distributions and LID may not be properly accounted for or fully exploited, and so there is likely more work that needs to be done here. (q.v. the LID model used in the two papers by Ma et al.)

W2) Although the authors make the case that ID regularization does lead to improvement in autoencoder-based applications, it is not clear whether they are sufficient to allow them to compete with state-of-the-art contrastive learning models. It would have been good to see this question settled with a more extensive experimentation.

The model proposed seems incremental based on regularization over well-established architectures. There are novelty concerns regarding the training process as well. Qualitative results to empirical findings also seem inadequate.

• The novelty seems incremental. The motivation of AE-IDC lacks insights. Specifically,
  • The Intrinsic dimension is not a new concept. It lacks insights to incorporate ID into AE/SAE by simply adding minimizing $ID(X) - ID(\hat X)$. There are plenty of works adopting diverse terms for AE.
  • The training paradigm is almost the same as the classical SAE[1]. It is not new.

​ Overall, the contributions may not be enough for ICLR.

• The improvements shown in experiments seem incremental. The improvement is foreseeable when some regularizations are incorporated into AE/SAE.
• The authors fail to clarify why ID is suitable for image data.

[1] Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion, JMLR, 2010.