Contrastive Implicit Representation Learning

Major

• Table 2: I’d like to see the average performance and standard deviations (in brackets next to the average performance) rather than the best accuracy for all runs. Reporting best accuracy is cherry-picking and not representative. Please change this accordingly. Why do you think the performance for contrastive learning is so poor compared to supervised learning? Do you have an explanation or at least an idea? The MNIST results look pretty bad.

• Section 4.2: Why do you think the learning rate has a large impact on the model’s data fitting ability? This seems to be brittle.

• Figure 3: I don’t see any useful structure emerging from t-SNE visualizations for CIFAR-10. The visualization indicate that the representations are almost entirely entangled. For MNIST it looks fine, but I think we are past MNIST in Computer Vision.

• Section 4.3: You write “The contrastive method surprisingly performs more closely to the supervised one on the MFN, compared to SIREN. This suggests that performing contrastive learning is more beneficial for certain architectures of neural fields.” I am not convinced that you can draw the latter conclusion. I am not surprised at all by this finding. The performance of SIREN is pretty strong for the supervised learning method. This is close to what I would expect for a good classifier to achieve on MNIST (CIFAR-10 is a bit weaker than what I would expect for a good classifier but it is not terrible). However, the performance of MFN using supervised learning is pretty poor. Contrastive seems to work poorly irrespective of whether you use SIREN or MFN, although it is even worse for MFN (honestly, the Contrastive + MFN results are one of the worst performances I have ever seen for MNIST and CIFAR-10). So, it is pretty obvious that contrastive and supervised learning performances are closer for MFN than they are for SIREN. Neither MFN nor contrastive seem to yield useful representations. So, it is expected that for MFN there's not a big difference between supervised and contrastive. The only useful representations that seem to emerge are the ones found by using supervised learning in combination with SIREN.

• Findings re augmentations: It is not great that most variance is attributed to the random seed “augmentations” and that they affect performance more than any of the other augmentations. Why do you think the initialization matters so much here?

Minor

• Why is the signal domain dimension $d = 2$? Is this because of height and width and each of the two defines a single dimension? This is not clearly defined anywhere. Although this might be trivial to you, a reader can only guess why the signal domain dimension $d = 2$. This is not great. There should not be a need to guess.

• In the Introduction you define a discrete signal as ${\{I_{i}\}}_{i=1}^{N}$. What is $I$? This is not defined anywhere. Again, I can only guess that you refer to an image by $I$. However, it may not be obvious to everyone. People use different notations for inputs. I usually use $x$ rather than $I$, even if the input is an image.

• There is a typo in the second line of subsection 1.1. It should read “feature extraction” and not “features extraction”. In reason one of subsection 1.1, you claim that “[...]. One key challenge of self-supervised methods is learning representations that generalize well across datasets of different shapes and image sizes.” I get the latter point. Different datasets have different image sizes. But what do you mean by differently shaped datasets and how is the “poor” generalization performance of contrastive learning across “differently shaped datasets” quantified? I’d be curious to better understand this.

• Section 2: “a dataset of implicit representation (NOTE: this should be plural and not singular I suppose) is simply a set $\{\theta_{j}\}$. $j$ seems to be an index. From where to where is this index running? First paragraph in section 2: Another typo. What is “eq. equation 2”? Should probably read “Eq. 2”, “eq. (2)”, or “Equation 2.”.

• Section 3.2: What do you mean by object identity? Could you elaborate?

The major weakness of this work is that it seems quite half-baked and shallow, and its contributions might be insufficient for a conference publication.

Regarding the proposed methods, the new ones that, as far as I know, did not appear in prior work are the geometric augmentations for INR, random seed augmentations, and the regularizer Eqn. (9). These ideas, while they do make sense, seem more like some initial thoughts rather than well-formulated methods. And the justification for the proposed methods are quite weak. Theoretically, there is no guarantee that these methods would work. Empirically, in Table 2, the performance of contrastive learning seems much worse than supervised learning.

Of course, a paper could be a good paper if it raises an important question that arouses the awareness of the community, even without providing a solution. But I do not think this work makes sufficient contributions in this direction either. First of all, learning representations for implicit representations is not a new problem. There is lots of work on this in the Nerf community. Second, I don’t think the authors catch the real central question. From my understanding, the central question is this: If images are represented in a format other than pixel values (that is, Euclidean), can one still perform contrastive learning? In fact, there are lots of representations of images, including Fourier representation, wavelet representation, and the neural representation studied in this work. The main challenge is can we find semantic-preserving data augmentations for each one of these representations. For example, for Fourier representation, it could be the case that high frequency components have little effect on the semantics, so altering these components could be one such augmentation. After reading this submission, I get the impression that contrastive learning is not very feasible for INR, mostly because the space of INR does not have a basis like the Fourier basis, so the aliasing problem seems very hard to solve.

Above I provided some suggested directions and possible ideas to the authors. There are of course other directions the authors could explore, such as the effect of the network architecture on the structure of the INR space. Also, it is totally fine if the authors choose to only focus on INR, but in that case I think the authors need to carry out a much more extensive theoretical analysis and/or empirical study.

• The experiments are only on CIFAR10 and MNIST which are small-scale. It's unclear if the proposed augmentations generalize to more realistic and impactful datasets such as ImageNet. Given that the goal of self-supervised methods are to scale to large datasets (LAION, JFT, Instagram-2B, etc.) which are weakly or not labeled, I think it's reasonable to show results on at least ImageNet.

• The figures aren't particularly informative. For those unfamiliar with MFN's, figure 1 is difficult to interpret. A figure illustrating the main idea of the paper would be useful. Also the table of hyper-parameters (Table 1) does not convey much information. It may be better in the appendix.

Minor Comments:

• It would be more readable if the SIREN and MFN acronym in the abstract are introduced before using the term.
• The table of hyper-parameters (Table 1) is not that informative, especially since it's the first table of the paper. It may be better in the appendix.