Gaussian Masked Autoencoders

We thank the reviewer for comments on our paper where the “idea is novel”, presented in a “well written paper and wide range of experiments which are reproducible”. Here we answer the questions the reviewer is asking, and we are happy to answer more questions or run more experiments for the rebuttal. We will also open source the code and the models.

Comparison with contrastive learning methods.

We thank the reviewer for the suggestion. We will add more baseline contrastive approaches in Table 2a. We will also explore more tasks. For example, we have already added a depth estimation task, and we show GMAE performs better than DINOv2 in depth estimation. We want to highlight that, even after zero-shot capabilities were introduced, the model does not lose its encoder representation power.

“What qualities does Gaussian splatting have to help MAE? Can Gaussian splatting be replaced with NeRF or other 3D representations?”

We were interested in learning from large scale images, and MAE is proven to be the best approach to learn good semantic representations. On the other hand, Gaussians are good way to learn structure/geometry. A combination of these two approaches allowed us to learn good semantic representations (as shown in imagenet, coco experiments), as well as some geometry based on 2.1d representations (layering and edge detections).

Regarding NERF vs Gaussians, the more important property we see from Gaussian Splatting is that it's an unstructured sparse representation that automatically allocates more resources on finer details. Neither of the NeRF variants come with this property. Though some other 3D representations such as point cloud or meshes also allow spatially variant sparsity, they are much harder to optimize than 3D Gaussians. So we see Gaussians are a great way to represent not only 3D content, but also (layered) 2D content in the machine learning context.

Official Response by Author for the Reviewer Zk2V

We thank the reviewer for comments that mention that our “idea is novel and critical to the computer vision community”, and that acknowledge our well written paper and wide range of experiments. Here we answer the questions the reviewer is asking, and we are happy to answer more questions or run more experiments for the rebuttal and we fix the typos in the paper.

“The evaluation datasets and used baselines seem to be a bit outdated”

For edge detection, even SAM is evaluated on the BSD500 dataset, and for the segmentation tasks, we are following the protocol and the code in MAE-VQGAN. We have added new results on NYU-scenes on depth estimation. We appreciate the suggestion to compare to DINOv2 and added it as an additional baseline. We will add more tasks which are relevant for this during this discussion period.

“There are no failure case examples to justify the possible future work of GMAE method.”

Thanks for bringing up this point. We did share unfiltered, randomly chosen samples from our models, in appendix figure 12. A few failure modes we would like to list are a) in the cases of layering, sometimes the layers are still not fully disentangled from colors. b) another limitation of this work is the use of L2 loss, which tends to give results which are blurry, compared to, for example, diffusion style loss. c) our model suffers from a feedforward vs optimization tradeoff, optimization approach like gsplat training would give better results while being slow, while gmae on other hard, being a feedforward fast approach, but giving not the best reconstruction results. d) the number of gaussians was another main limitation of our work, however, based on the new results, we have fixed this by learning additional residual gaussians. Now we can learn up to 4096 gaussians.

“How to get the query tokens? What could these tokens be?” Query tokens are learned tokens, which were trained end-to-end during the training of the encoder and the decoder, and we initialize them randomly with zero mean and 0.02 variance. At test time, The query tokens and the latent tokens from the encoder are concatenated and passed through the decoder, and the query tokens are projected into gaussians for rendering. We will make this point more clear in the updated manuscript.

We thank the reviewer for comments on our intriguing idea, meticulously designed model, and satisfactory zero-shot capabilities. Here we answer the questions the reviewer is asking, and we are happy to answer more questions or run more experiments for the rebuttal.

“Demonstrating a positive impact on image generation would significantly enhance the paper’s contributions.”

We agree! However, our current goal is to show the representation capabilities of the self-supervised models rather than their generation capabilities, which would require re-thinking the model, for example, using DiT models. While this is not the scope of the paper, to archive this number of gaussians was one of the main bottlenecks, which the reviewer also mentioned. Based on the new results we have fixed this problem and were able to scale up to 4096 gaussians. This could be a potential tokenizer to train generative models now, however since that is beyond the scope of this work, we will leave it for future work.

“If increasing the Gaussian count presents a bottleneck, this limitation could hinder its application in image generation tasks.”

We agree with the reviewer, that the limit on the number of gaussians is a limitation in the current model, and the current design we are limited by memory. We tried to solve this problem by learning more gaussians step by step, as residual gaussians. First, we pretrained the model with 256 Gaussians, and after that, we initialize an k MLP layers, to project changes in 14 features to k*256 Gaussians. These mlp layers essentially learn the small changes from the main 256 gaussians. At the start these mlp layers are initialized with zero weights, hence they don't affect the reconstruction for the original 256 gaussians.

With this setting, we were able to train up to 4096 gaussians, but not limited (we will update here with more gaussians), since it is not limited by memory anymore. Below we show the reconstruction FID as we increase the number of gaussians step by step. Finally, we also fine tune the model (4096*) without masking and on full reconstruction, and it achieves 18 rfid, without perceptual loss or vae.

“Would be more informative if the authors provided the absolute compute times for both MAE and GMAE”

For MAE training on V100 gpus, takes on average time: 0.6471 seconds and Standard deviation: 0.0209 for 10 samples for forward and backward pass vs GMAE takes on average a mean time: 0.7044 seconds and a standard deviation: 0.0053 for 10 samples including forward, rendering, and backward pass. Which it added only a small overhard to the training, which recent advances in gslpat optimizations can also be incorporated to give faster training.

“Arranging the visualization of Gaussian layers from shallow to deep depths”

Thanks for this suggestion, we have added figures from shallow to deep in the appendix on lots of unfiltered samples.

We thank the reviewer for comments on our well written paper, high quality visualizations, and good quality of a scientific paper. Here we answer the questions the reviewer is asking, and we are happy to answer more questions or run more experiments for the rebuttal.

“The method looks very unnatural and simply combines 2 popular ideas: 3d gaussians and MAEs.”

We agree with the reviewer, these are two bit orthogonal approaches. However, we'd like to cast the comment with positive light, being very unnatural meaning that we are doing something novel, and we showed that Gaussian and MAE can be integrated into the same framework. If simple combination can already show interesting results, it would be valuable to explore and share these findings with the community.

our combination is unique and we showed some interesting results. We were interested in learning from large scale images, and MAE is proven to be a best approach to learn good semantic representations, on the other hand, Gaussians are good way to learn structure/geometry. A combination of these two approaches allowed us to learn good semantic representations (as shown in ImageNet, COCO experiments), as well as some geometry based on 2.1d representations (layering and edge detections).

In addition to this, our new results on depth estimation shows it is more competitive than dinov2, and also our new results on scaling the number of gaussians shows this could be helpful with image generation tasks.

“Zero-shot capabilities are not convincing”

Our goal is to demonstrate that these capabilities emerge purely from self supervised pre training objectives, without any task specific design choices. None of the results were explicitly trained for the zero-shot tasks. While we agree that task-specific models which were trained for these tasks might get better results, we only showed few zero-shot capabilities by only training on self-supervised objectives. We believed there might be more tasks which can be explored, from our models. We only showed a few examples, but since these are tasks the model was not optimized for, there might be more tasks that can be done with our models. Could the reviewer please clarify regarding "generative methods" or "generative multi-plane images", which shows zero-shot capabilities also emerge while the models are not specifically pre-trained for those tasks.

“I would hope to see is having some 3D capabilities”

it would be impossible to get full 3D representation from a large scale collection of 2D images from different scenes. We aim to get to some level higher than 2D and learn a 2.1D representation which would allow us to learn layering of the objects and scene. Hopefully, with addition of videos to the training data, we may be able to learn some slightly higher than 2.1D representations. We leave this exploration for future work.

“What exactly is the main advantage of the proposed model? For which use-case would one realistically choose to use it?”

We agree the tasks we have are not fully utilized, and if we care about a single zero shot task, we can simply use the best methods available there to get better performance. But rather, this work shows there is another way to train self-supervised models and how to get zero-shot capabilities from these models. From this we showed, a subset of zero-shot capabilities emerge from GMAE models, but this is not limited, and this opens more possibilities to explore more zero-shot capabilities in self-supervised models as a meta task.

In addition to this, we also showed that in depth estimation case GMAE performs better than dinov2 and same as mae self-supervised models. We also showed a new way to increase the number of gaussians by 16x factor, by learning residual gaussians. This could be used as a better initialization for gaussian splatting, or for generative models.

Finally, we wanted to address that, equipped w/ Gaussians, we can unlock the use of decoders after MAE pre-training. Traditional SSL methods only care about the encoders; we show that with the Gaussians in the decoder, we can actually use the Gaussians to do more tasks (zero-shot). Of course these are some initial positive signals, but we believe these signals are interesting and worthy of sharing to the community.

“How much slower each training iteration has become compared to MAE?”

For MAE training on V100 gpus, takes on average time: 0.6471 seconds and Standard deviation: 0.0209 for 10 samples for forward and backward pass vs GMAE takes on average a mean time: 0.7044 seconds and a standard deviation: 0.0053 for 10 samples including forward, rendering, and backward pass. It added only a small overhead to the training, which recent advances in gslpat optimizations can also be incorporated to give faster training.

We thank the reviewer for their comments on our interesting approach, wide variety of experiments and clear writing. Here we answer the questions the reviewer is asking, and we are happy to answer more questions or run more experiments for the rebuttal.

“The number of Gaussians used in GMAE is significantly lower than reconstruction methods”

We agree with the reviewer, that the limit on the number of gaussians is a limitation in the current model (it's a limitation as this is the first time someone combines Gaussians to a representation learning based framework), and the current design we are limited by memory. We want to point out that we are not focused on reconstruction quality here and the focus is on the meaning of the learned Gaussians. However, here we modify the architecture to show the potential of further increasing the number of Gaussians.

We tried to solve this problem by learning more Gaussians step by step, as residual gaussians. First, we pretrained the model with 256 Gaussians, and after that, we initialize an k MLP layers, to project changes in 14 features to k*256 Gaussians. These MLP layers essentially learn the small changes from the main 256 gaussians. At the start these MLP layers are initialized with zero weights, hence they don't affect the reconstruction for the original 256 gaussians.

With this setting, we were able to train up to 4096 gaussians, but not limited (we will update here with more gaussians), since it is not limited by the memory size anymore. Below we show the reconstruction FID as we increase the number of gaussians step by step. Finally, we also fine tune the model (4096*) without masking and on full reconstruction, and it achieves 18 rFID, without perceptual loss or vae.

“Any benefits to manually reducing redundant DoFs when training such models?”

The 14 DoFs includes 3 for location, 3 for color, 4 for rotation as quaternion, 3 for scale and 1 for opacity. If we treat all GSs as 2D instead of 3D living on the image plane, then we could reduce the DoFs to 11 (2 for location, 2 for rotation and 2 for scale). But in this case we lose the ability to model the image as "layers" because all GSs effectively placed at the same depth level. Since, our aim is to have this extra degree of freedom to allow the model to place gaussians at different depth levels, so that we can get these emergent zero-shot capabilities. We will train a model with 2DGS, and get back before the discussion period ends.

“it would be interesting to compare this model with MAEs in a depth-prediction”

We thank the reviewer for this valuable feedback, and we have evaluated our models on depth estimation tasks. Please find the depth estimation results below, on NYU depth estimation tasks, GMAE outperforms dinov2 models and perform the same as MAE or slightly better.

“How well does a decoder generalize to Gaussian counts other than the one it was trained on?”

At test time we can only infer with a fixed number of gaussians with the current design, but we plan to explore training the models on Matryoshka style loss, which can allow us to use variable number of gaussians at test time and gives coarse to fine reconstructions as we increase the number of gaussians. We will add a discussion regarding test time scaling of the number of gaussians in the paper.

We thank the reviewer for comments on the ”paper is original and interesting”, “paper has potential to be significant to the community” and “paper is presented clearly and laid out well”. Here we answer the questions the reviewer is asking, and we are happy to answer more questions or run more experiments for the rebuttal.

Related works

We agree that our proposed mid-level representation is but one of many existing options, with many more to come. While we cited several of these related works, we will also make sure to modify the introduction and related works to further highlight this fact in the updated version of the submission.

We agree with the reviewer that there are various options for mid-level representations. We thank the reviewer for pointing out Superpixel Sampling Networks (SSN), and we will discuss them in the related works section. However, there are a few differences we would like to point out. While there is a similarity between gaussians and superpixels as intermediate representations, unlike SSN, our models are trained for self-supervision without any task specific data. Edges and segments are extracted from our model, rather than trained for end-to-end as in Superpixel Sampling Networks.

Our approach differs from SSN or Slot attention in several ways. Unlike SSN or Slot attention, we don't utilize any specific loss function, and we simply use L2 reconstruction loss. In terms of architecture, we don't add any specific changes to the model. In addition to this we also show that our method can learn representations that are both good for recognition and detection as well as mid-level zero-shot problems.

We also cited Bar et al. in our main paper, discussed it in the related works section, and compared to it in table 3. We found that our method outperformed Bar et al. in figure ground tasks. In addition to that, bar et al only produce discrete tokens, and it is hard to reason about or manipulate the discrete token, since they don't have any explicit representation.

"Nonetheless, we have shown that one no longer has to choose between pixels and latent representations for visual modeling." — based on the additional related works discussed above, we see this can be misleading and we will remove this in the paper.

“Why is the gaussian representation better?”

We believe that, GMAE adds extra capabilities while persevering the representation quality along the MAE axis. We explored new directions such as layering, segmentation, and edge detection. The rebuttal also includes new experimental results on metric depth estimation where GMAE performs better than dinov2 on NYU depth estimation. We also explored the generation quality of GMAE and with new changes to the architectures, we were able to train with 4096 gaussians and more, without memory issues. All these additional capabilities are due to the use of gaussians as an intermediate representation, and it that could open up new capabilities in the future. Below we address a few concerns regarding the GMAE model.

Slower: GMAE is not significantly slower than MAE. For MAE training on V100 gpus takes on average: 0.6471 seconds with standard deviation: 0.0209 for 10 samples for forward and backward pass. GMAE takes on average: 0.6944 seconds and a standard deviation: 0.0053 for 10 samples including forward, rendering, and backward pass. This adds only a small overhead to training. Recent advances in GSplat optimizations can also be incorporated to result in faster training.

GAN loss: The MAE github mentions that an addition of a GAN loss leads to better reconstructions. According to off-line communications with MAE authors, the additional adversarial training only improves reconstruction quality (the images become sharper), but it does not improve representation quality. Bar et al, uses discrete tokens from VQ-GAN and is trained with a GAN loss to have sharp reconstructions.

Frequency clustering in depth: “Individual instances of objects are not separated in any way” — this is true, and we do not claim this, as based on the training recipe it is unlikely that the model would be able to group based on instances. Our claim was “separate objects in the z direction”, this is layering rather than instance segmentations. We agree with the reviewer that these are not explicit segmentation rather, reasonable reconstructions of different layers.