Components Beat Patches: Eigenvector Removal for Robust Masked Image Modelling

Dear reviewer 1XJX,

Thank you for your review of our work. Please find below our answers to your questions:

Baseline performance

Our work proposes the masking of principal components rather than pixels. We view this contribution as a proof-of-concept, demonstrating that masking in the space of principal components can significantly enhance downstream performance and inspire further research in Masked Image Modeling. This perspective aligns with feedback from other reviewers, such as reviewer WJuv, who noted: "The idea is simple yet solid, even drawing intuitions from early research in image processing / Eigenfaces, which I can imagine inspiring future/new directions in representation learning”. With this goal in mind, we verify our claims by conducting experiments on a ViT-tiny for 800 training epochs on 5 datasets spanning across different domains (i.e., natural and medical images).

A ViT-tiny is a smaller-scale architecture of roughly 30.8M parameters (~3x and ~10x less than a Vit-B and Vit-L on which ImageNet1k is commonly evaluated). It is often used for datasets such as CIFAR10 and TinyImageNet. The following pointers (recently published) provide insights into the standard range of linear probing performance achieved for CIFAR10 and TinyImageNet datasets with MAEs with such architecture.

Table 1 in [2] reports a performance of 59.6% for CIFAR10 after 2000 training epochs with a Vit-tiny. Table 2 in [3] trains a Vit-tiny on CIFAR10 and TinyImageNet for 2000 and 1000 epochs, respectively, and shows a classification performance with a linear probe of 72.5% and 19.6%, respectively.

Beyond the architecture scale, fine-tuning is expected to yield significantly higher performance compared to using a linear probe. Table 2 of [4] presents fine-tuning results on a ViT-B architecture for CIFAR-10 and TinyImageNet, achieving image classification accuracies exceeding 90% and 60%, respectively. These observations collectively indicate that lower linear probing scores on a ViT-tiny are anticipated and should not be interpreted as poor performance of MAE on smaller images.

Additionally, we ensured that only minimal modifications were made to the original MAE codebase [1], maintaining consistency between the original approach and our reported results. Our codebase is included in the supplementary materials submitted with the manuscript. We hope these explanations clarify why the baseline scores (MAE) reported in our work are lower than the current state-of-the-art for these datasets. As highlighted by other reviewers --- “The evaluation strategy is solid and the use of SOTA baselines is good.” (reviewer 5bnU) ---, we believe our empirical validation provides strong evidence that the space of principal components is a meaningful alternative to the image space for self-supervised representation learning.

We welcome any additional questions you may have regarding our implementation of the baselines.

[1] https://github.com/facebookresearch/mae

[2] Zhang, Qi, Yifei Wang, and Yisen Wang. "How mask matters: Towards theoretical understandings of masked autoencoders." Advances in Neural Information Processing Systems 35 (2022): 27127-27139.

[3] Zhang, Kevin, and Zhiqiang Shen. "i-mae: Are latent representations in masked autoencoders linearly separable?." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[4] Mao, Jiawei, et al. "Masked autoencoders are effective solution to transformer data-hungry." arXiv preprint arXiv:2212.05677 (2022).

Scalability of PMAE to large datasets

As PMAE relies on PCA, extending our experimental setup to larger-scale datasets would require the computation of their principal components. As rightfully mentioned by reviewer WJuv, the cost of this operation scales cubically with the data dimensionality (i.e., $O(d^2n+d^3)$ for a dataset with $n$ samples of dimensionality $d$). To lower this cost, especially for larger images like ImageNet1k, future work could consider other more cost-effective data transformations (please see the general answer to reviewers for a more thorough discussion on future directions for scalability).

While these additional costs currently limit the applicability of PMAE to larger datasets, our work shows substantial performance gains obtained with PCA on a range of medium-sized datasets. These findings lay the groundwork for the exploration of other image transformations, perhaps more cost-effective, as mappings to a latent space in which masking is performed. Besides, we do not intend to claim that PMAE is superior to MAE in all scenarios, including on large-scale and very high-dimensional datasets. Rather we view our work as a proof of concept that masking in a learnt latent space can be beneficial for self-supervised masked image modelling. We provide clear evidence of this for several medium-sized settings, but agree with the reviewer that further work (e.g., transformations beyond standard PCA) is needed to scale our alternative masking strategy to more complex scenarios.

Thanks to your feedback, we updated section 8 of the manuscript to further discuss the scalability of our approach as follows: “Other off-the-shelf non-linear transformations, such as the Fourier transform, Wavelet transform, Kernel Principal Component Analysis, or Diffusion Maps, represent alternative candidate transformations. Future research should explore whether the properties of these spaces provide comparable or additional advantages over PCA. [...] A particularly appealing aspect of some of these methods (e.g., Fourier & Wavelet transforms and Diffusion Maps) is the use of fixed bases, which could eliminate the computational overhead of PCA---whose cost scales cubically with the data dimensionality--- and improve scalability to larger datasets.”

Thank you again for your feedback. We welcome any additional suggestions, questions, or requests for information and encourage further discussions.

Finally, please refer to the general answer to reviewers for additional results added during this rebuttal that further support our claims.

Dear reviewer 5bnU,

Thank you for your thorough review of our work! Please find below our answers to your questions:

Going beyond PCA

We agree that many other invertible transformations, including non-linear transformations, could serve as interesting alternatives to PCA. We see our work as proof of concept, showing that masking in latent space instead of the observation space consistently leads to substantial performance gains across datasets. Building on these results, exploring other data transformations, such as the Fourier or wavelet transform, represents promising directions for future research with the potential for additional benefits on downstream tasks.

Following a suggestion from reviewer WJuv, we have conducted preliminary research using Kernel PCA in place of PCA on the CIFAR10 dataset. These additional results were incorporated into the appendix in section A.6.4.

Kernel PCA [1] is a non-linear data transformation, which differs from PCA by performing the spectral decomposition in a latent space in place of the space of observations. The data is first mapped to a high-dimensional space using a kernel function — we chose the Radial Basis Function (RBF) kernel — before spectral decomposition is performed.

Table 1 below reports the image classification accuracy using a linear and MLP probe for CIFAR10 for a standard MAE, PMAE, and KMAE, which relies on Kernel PCA. Results show performance gains of 13.3 and 5 percentage points brought by KMAE over MAE and PMAE, respectively, when using a linear probe. These preliminary findings suggest that masking in latent space can be extended beyond PCA and lead to further empirical gains.

Please note that scores for PMAE reported in table 1 are computed following the alternative objective described in “Additional results supporting our claims” in the general answer to reviewers.

Table 1: Linear and MLP probe accuracy for MAE, PMAE, KMAE with best masking ratios for each method. Bold numbers refer to the best scores, italic to the second best. * refers to our work.

Based on your feedback and these interesting additional results, we have improved our discussion in section 8 as follows: “Other off-the-shelf non-linear transformations, such as the Fourier transform, Wavelet transform, Kernel Principal Component Analysis, or Diffusion Maps, represent alternative candidate transformations. Future research should explore whether the properties of these spaces provide comparable or additional advantages over PCA. Preliminary results on Kernelized PCA, presented in appendix A.6.4, demonstrate performance gains over PMAE, motivating further exploration.”

[1] Schölkopf, Bernhard, Alexander Smola, and Klaus-Robert Müller. "Kernel principal component analysis." International conference on artificial neural networks. Berlin, Heidelberg: Springer Berlin Heidelberg, 1997.

Dear reviewer WJuV,

Thank you for your thorough review of our work! Please find below the answers to your questions.

Scalability of PMAE

As you correctly point out, the cost of eigendecomposition increases cubically with the data dimensionality, which poses challenges for very large datasets. Although computing a subset of principal components would help reduce these costs, its effects on the proposed masking strategy remain unclear. Indeed, performing a low-rank approximation would lead to a drop of the eigenvectors with lowest eigenvalues which previous work [1] showed carries features meaningful for perceptual tasks.

Instead, future work could consider other data transformations like the Fourier transform [2], Wavelet transform [3], or Diffusion Maps [4] in place of the PCA. Notably, the Discrete Cosine Transform (DCT [5]), closely related to the Fourier transform, could be a viable cost-effective alternative to PCA. The DCT can be described as a discrete Fourier transform operating on real numbers only and is an approximation of PCA [6]. Its advantage lies in its cost. Unlike PCA, it relies on a fixed predefined basis and would thereby lower computational costs and help with scalability.

Combined with our empirical findings on the benefits of the principal component space for image masking, these observations suggest that exploring a broader range of image transformations could open promising research directions while lowering computational costs.

Thanks to your feedback and that of other reviewers, we have updated our section 8 to further discuss the scalability of PMAE to larger datasets as follows: “Other off-the-shelf non-linear transformations, such as the Fourier transform, Wavelet transform, Kernel Principal Component Analysis, or Diffusion Maps, represent alternative candidate transformations. Future research should explore whether the properties of these spaces provide comparable or additional advantages over PCA. [...] A particularly appealing aspect of some of these methods (e.g., Fourier & Wavelet transforms and Diffusion Maps) is the use of fixed bases, which could eliminate the computational overhead of PCA---whose cost scales cubically with the data dimensionality--- and improve scalability to larger datasets.”

[1] Balestriero, Randall, and Yann LeCun. "How Learning by Reconstruction Produces Uninformative Features For Perception." Forty-first International Conference on Machine Learning.

[2] Bracewell, R. (1986). The Fourier Transform and Its Applications. McGraw-Hill.

[3] Daubechies, I. (1992). Ten Lectures on Wavelets. SIAM.

[4] Coifman, R. R., & Lafon, S. (2006). Diffusion Maps. Applied and Computational Harmonic Analysis, 21(1), 5–30.

[5] Ahmed, Nasir, T_ Natarajan, and Kamisetty R. Rao. "Discrete cosine transform." IEEE transactions on Computers 100.1 (1974): 90-93.

[6] Sanchez, Victoria, et al. "Diagonalizing properties of the discrete cosine transforms." IEEE transactions on Signal Processing 43.11 (1995): 2631-2641.

Dear reviewer tnEx,

Thank you for your review of our work.

We would like to clarify some points regarding the proposed work because your main point of criticism is based on a misunderstanding of a fundamental aspect of our work.

Principal component analysis (PCA) finds components that maximize variance in the projected space, resulting in uncorrelated components (i.e., no linear relationship between principal components). However, this does not imply that the principal components are independent, as uncorrelatedness does not imply independence. In contrast, independent component analysis (ICA), which our work does not rely on, seeks to separate data into statistically independent components. In our opinion, it is reasonable to consider that natural or medical images are produced by a non-linear generative process. Consequently, it is unlikely that all principal components of natural or medical images would be independent.

Our empirical results (c.f., Table 1 in our manuscript) show performance scores way above those of random predictions, which would not be possible if the input image and reconstruction target were to be independent. Examples of visible and masked principled components, provided in Figure 3 (blue) of the original manuscript, also show visual evidence that disjoint sets of principal components are not independent of one another (i.e., the animal contours are present in both the input and target images in Figure 3 (blue)).

Finally, we would like to conclude by emphasizing that other reviewers have recognized the soundness of our approach: “The idea is simple yet solid, even drawing intuitions from early research in image processing / Eigenfaces, which I can imagine inspiring future/new directions in representation learning.” (reviewer WJuv); “The idea of using PCA as an invertible transformation into a latent space is sensible” (reviewer 5bnU).

Further discussion regarding the scaling of our approach to larger-scale datasets is provided in the general answer to reviewers.

Thank you again for your feedback. We hope we could address your concerns. In light of these clarifications regarding some fundamental aspects of our work, we kindly ask you to reconsider your evaluation.

We welcome any additional suggestions, questions, or requests for information and encourage further discussions.

Additional results supporting our claims

Finally, we are pleased to report additional empirical results supporting our claims that the space of principal components is a meaningful masking space. We explore a simple change of our original setup, presented in Figure 1 of our manuscript, and observe larger performance gains compared to the results originally reported.

In the original manuscript, we applied the reconstruction loss directly in the image space, as depicted in Figure 1. Our training objective, presented in equation 3.1, minimizes the Euclidean distance between the decoder’s output and the masked principal components projected back to image space. A simple alternative to this learning objective is to minimize the Euclidean distance in PC space between the masked principal components and the reconstruction of the masked principal components. This alternative to equation 3.1, now presented in appendix A.6.3 and equation A.1, further improves downstream classification performance as presented in tables 2 & 3 below.

Table 2: Linear probe accuracy for MAE, PMAE, for CIFAR10, TinyImageNet, BloodMNIST, DermaMNIST, PathMNIST for optimal masking ratios. Bold numbers refer to the best scores.

Table 3: MLP probe accuracy for MAE, PMAE, for CIFAR10, TinyImageNet, BloodMNIST, DermaMNIST, PathMNIST for optimal masking ratios. Bold numbers refer to the best scores.

Tables 2 and 3 show an average performance gain of 9.6 percentage points over the MAE baseline with a linear probe. In comparison, our original results showed an average gain of 6.6 percentage points over the MAE baseline. These results further support our claims by showing that the modeling of masked principal components constitutes a strong learning paradigm.

We welcome any additional suggestions, questions, or requests for information and encourage further discussions. Once again, we thank all the reviewers for their time and effort in evaluating our work.

Scalability of PMAE (reviewers WJuv, tnEx,1XJX,5bnU)

Our work aims at showing the potential of the PC space for Masked Image Modeling (MIM) learning paradigms. We support our claims by showing the superiority of the space of principal components over the space of observations. We report substantial performance gains on 5 datasets with image resolutions ranging from 32x32 to 64x64 and training sample size ranging from 7k to 100k samples. These findings act as a proof of concept showing the appeal of PCA for MIM and suggest that exploring other image transformations (i.e., Fourier Transform [1], Wavelet transform [2], Diffusion Maps [3], Kernel PCA [6],...) is an interesting direction for further research.

As PMAE relies on PCA, extending our experimental setup to larger-scale datasets requires the computation of their principal components. As rightfully mentioned by reviewer WJuv, the cost of this operation scales cubically with the data dimensionality (i.e., $O(d^2n+d^3)$ for a dataset with $n$ samples of dimensionality $d$).

To lower this cost, future work could consider other data transformations like the Fourier transform [1], Wavelet transform [2], or Diffusion Maps [3]. Notably, the Discrete Cosine Transform (DCT [4]), closely related to the Fourier transform, could serve as a viable and cost-effective alternative to PCA. The DCT can be described as a discrete Fourier transform operating on real numbers only. In addition, the DCT is an approximation of PCA [5]. The advantage of the DCT lies in its cost. Unlike PCA, it relies on a fixed predefined basis and would lower computational costs and help scale up the proposed approach to even larger datasets.

While PMAE is currently limited to mid-scale datasets, our work lays the groundwork for exploring new masking spaces for self-supervised learning by highlighting substantial performance gains obtained with PCA on various datasets.

We have updated section 8 of the manuscript to further discuss the scalability of our approach as follows: “Other off-the-shelf non-linear transformations, such as the Fourier transform, Wavelet transform, Kernel Principal Component Analysis, or Diffusion Maps, represent alternative candidate transformations. Future research should explore whether the properties of these spaces provide comparable or additional advantages over PCA. [...] A particularly appealing aspect of some of these methods (e.g., Fourier & Wavelet transforms and Diffusion Maps) is the use of fixed bases, which could eliminate the computational overhead of PCA---whose cost scales cubically with the data dimensionality--- and improve scalability to larger datasets.”

We thank the reviewers for their valuable input, which helped improve our work.

[1] Bracewell, R. (1986). The Fourier Transform and Its Applications. McGraw-Hill.

[2] Daubechies, I. (1992). Ten Lectures on Wavelets. SIAM.

[3] Coifman, R. R., & Lafon, S. (2006). Diffusion Maps. Applied and Computational Harmonic Analysis, 21(1), 5–30.

[4] Ahmed, Nasir, T_ Natarajan, and Kamisetty R. Rao. "Discrete cosine transform." IEEE transactions on Computers 100.1 (1974): 90-93.

[5] Sanchez, Victoria, et al. "Diagonalizing properties of the discrete cosine transforms." IEEE transactions on Signal Processing 43.11 (1995): 2631-2641.

[6] Schölkopf, Bernhard, Alexander Smola, and Klaus-Robert Müller. "Kernel principal component analysis." International conference on artificial neural networks. Berlin, Heidelberg: Springer Berlin Heidelberg, 1997.

We greatly appreciate your words of encouragement and your positive opinion of our work. Thank you again for your time and effort in reviewing our work!