Self-supervised Representation Learning from Random Data Projectors

Thank you for your review and comments about our method’s good performance. We can discuss each of your points.

W1: Despite the good performance of the experiments, what has been learned from the latent space using the scheme introduced in this paper, even intuitively?

While we do not have a complete theoretical explanation for the success of learning from randomness, we can offer you some indication of an interesting direction. Random orthogonal projections on high dimensional spaces have been studied in mathematics and found to have interesting properties. One result, known as the Johnson-Lindenstrauss lemma [A], says that a set of points on a high-dimensional Euclidean space can be embedded into a lower-dimensional space such that distances are preserved. It has been observed that distances are still well-preserved when the embedding is a random orthogonal projection. This rigorous mathematical setting loosely mimics the practical setting of our paper - where high-dimensional data is projected into a low-dimensional space via a randomly initialized network. While the network does not produce an orthogonal embedding, it may still approximately preserve the distance between datapoints. This information can then be leveraged by the representation learner. We anticipate that the research community will be interested in exploring more theoretical explanations after publication.

W2: It says in the paper that g^(k)(x) uses the same architecture design of f_theta. Even with random initialization, it may still follow a certain distribution family. How would changing this architecture affect the learning?

There is literature (e.g. [B]) discussing what distributions are produced by randomly initialized networks, so we certainly agree with your point. We did not have space or capacity to give a full analysis of the types of distributions produced by each of the networks we used, but we hope that members of the community will find that direction interesting to pursue.

We found it sensible to use random projectors that share a similar architecture to the representation model since researchers have already determined classes of network architectures that have good inductive biases for extracting useful features. This information dictates the choice of representation model, and we reuse that information in choosing the projector architecture. In principle one could use any projector, but in preliminary experiments we found that matching inductive biases was beneficial.

W3: The paper only exams the accuracy on the classification task for frozen representations. Nonetheless, a good representation could used for various purposes, i.e., manipulation of each dimension in the latent space for generating new data, understanding the essential dynamics of the system in physical models and time series data. How could this strategy be applied to scenarios beyond classification?

As with any SSRL model, training is done without reference to any specific downstream task. Evaluation is commonly performed on classification tasks since class labels are widely available, but it would be possible to predict any kind of label, such as segmentations or depth in image data. On tabular datasets which were our main focus, there are not many examples of downstream tasks other than classification for which standardized datasets are publicly available. However, please see our general comment for a new experiment on transfer learning.

W4: Although the fact that choosing good augmentations usually requires domain knowledge, in many of the application mentioned, such knowledge could be partially obtained using physical intuition. By comparing with models under random corruptions doesn't seem to be a fair comparison.

We agree that with time and effort, it would be possible to produce useful semantic-preserving augmentations for most datasets, however our focus is on circumventing that process entirely. The advantage of LFR is that one does not need to put any effort into choosing good augmentations. In your last sentence we believe you are referring to one of the baseline methods, SCARF (Bahri et al., 2022), reviewed in Section 4.2. We believe that comparing to SCARF as one of the baselines is fair, since the random corruptions are exactly what (Bahri et al., 2022) proposed to use. We note that this is the only baseline that uses random corruptions. DACL for instance uses mixup for augmentations, while STab uses stochastic regularizations of networks.

[A] Freksen, “An Introduction to Johnson-Lindenstrauss Transforms” 2021

[B] Roberts et al., “The Principles of Deep Learning Theory”, Cambridge University Press 2022

We are glad that you appreciated the simplicity and generality of our approach, thank you for your feedback. We will go through each of your points.

W1: The main weakness for me is inability to compare with the existing method on well-known datasets in computer vision tasks. We have 3 x time series, 3 x tabular, and 1 x image - where Kvasir is not commonly used dataset.

We must disagree on one aspect - Kvasir is in fact widely used for benchmarking applications of ML to healthcare with over 1000 combined citations [Pogorelov et al. 2017] [A]. We used it as an example of the differences between domains (within the image modality), and how natural image augmentations are not necessarily semantic-preserving for medical images. This is a core motivation of our paper - moving towards domain-agnostic SSRL methods - and was not picked based on results.

As to your interest in seeing results on natural image data as well, and based on multiple requests from reviewers, we have rerun our experimental pipeline on CIFAR10. The results and discussion are in the general comment. In summary, LFR (using no augmentations) does not outperform SimCLR (using the full suite of augmentations) on natural image datasets. The same can be said for several domain-agnostic approaches including DIET and DACL.

W2: There's no evaluation on the different downstream task for the learned representation. We do not know how the learned representation generalize, and what if we only memorize patterns that then can be then useful for the lin. Evaluation.

Again, many reviewers had the same thoughts, so we have addressed your idea about transfer learning in our general comment. We originally did not include transfer learning experiments because they are not straightforward in the tabular and time-series domain. However, we did find a suitable setting from prior literature and applied LFR to it.

W3: We don't know the final computational overhead of the initialization and comparison to any SSL method. In the appendix we have total time spent for a particular dataset (e.g. Kvasir V100 1095 GPU hours for all experiments). Would be better to know the comparison between SimCLR/SimSiam vs LFR.

The initialization and selection of random projectors is a minor part of overall training time with LFR. As a concrete example, on the Income dataset training LFR for 100 epochs took 849.21s on our machine, while the DPP selection took 3.01s.

In terms of model training, SimCLR took around 9 hours on the Kvasir dataset, while LFR completed in just 2 hours on the same machine. Please see the general comment for a discussion of this result and our analysis of computational costs.

W4: 2 time series datasets (HAR, Epilepsy) and tabular Income&HEPMASS have already good accuracy on the randomized init. What is intresting, some methods are below that (HAR - Autoencoder).

The randomly initialized encoder baseline is intended to benchmark the improvement given by the learned representations. If the downstream task is easy, or the input features are already in a useful representation, then linear evaluation could perform well on any representation, even a randomly initialized one. Methods that do not outperform Random Init on linear evaluation are not providing more useful learned representations. In Table 1 there are various instances where an SSL method underperforms this baseline, possibly because the augmentations it uses are not well suited for the domain. However, LFR, being domain-agnostic, is able to learn more useful representations in every setting.

We are pleased that you found the motivation of our methods to be of great importance and our solution to be sensible suggestion. We will try to address each of your points.

W1: The experimental section does not include any medium/large scale dataset.

We disagree that there are no large-scale datasets used in our experiments. Please see Table 3 in Appendix C.1 for a summary of datasets and their sizes. We point out that MIMIC-III has a training set with over 2.5M datapoints, and HEPMASS consists of 7M. Meanwhile, our medical image dataset Kvasir consists of images at 80x100 resolution which is a fair bit larger than something like CIFAR10.

In the general comment, we have added experiments on two additional datasets - CIFAR10 for the natural images domain, and a fault diagnosis dataset from (Eldele et al. 2021).

W1/Q1/S1: Perform transfer learning experiments to support the main hypothesis of the work.

We appreciate this suggestion and that you think such results would “strongly reinforce the claim of the paper”. We originally did not include transfer learning experiments because they are not straightforward in the tabular and time-series domain. However, since it was a common request, we did find a suitable setting from prior literature and applied LFR to it. The results are discussed in the general comment.

W3/Q2: More discussion about the pros/cons of using random projections would be valuable.

We can provide some additional thoughts on the pros and cons of learning from random projections.

While we motivate our work with reference to domains where semantic-preserving augmentations are difficult to craft, there certainly are other domains where they do exist (e.g. natural images). In those cases, LFR does not make use of the expert domain knowledge that is available. This could lead to suboptimal performance, as indicated by our new results on CIFAR-10 in the general comment. One way to improve LFR would be through incorporating this domain knowledge when available.

LFR also requires initializing many networks for random projectors, but then discards many of them when selecting for diversity. This could be seen as wasteful, and the method could be improved if there were a good way to initialize more diverse networks to begin with, but we note that these are small networks, and that the entire process takes a negligible computational time compared to SSRL (3.01s to initialize and select for diversity, vs 849.21s to train).

In terms of pros, we have tried to highlight these throughout the paper. LFR can be used on any domain or modality. LFR has better performance across tasks where semantic-preserving augmentations are not well-developed, including important domains like medical images (Table 2). LFR can be more computationally efficient than methods like SimCLR and SimSiam that use multiple passes through the large encoder and CPU-based augmentations (see general comment).

Q3/S2: Evaluate in a setting where classic methods perform at their best. A natural images dataset is required for this paper for acceptance.

This was a common request amongst reviewers, so we have provided discussion and experimental results on CIFAR10 in the general comment. In summary, LFR (using no augmentations) does not outperform SimCLR (using the full suite of augmentations) on natural image datasets. The same can be said for several domain-agnostic approaches including DIET and DACL.

Q4: Could LFR be applied to Transformers?

We believe it is possible to use LFR with transformers, but have not been able to implement this idea in the discussion period. Hopefully the community is able to explore these ideas upon publication. As a brief outline, one could modify contrastive approaches like SimCSE [A] to use LFR by replacing how SimCSE samples positive pairs. Using a pre-trained transformer to transform text inputs to representation space, random projections can be applied instead of sampling pairs, and then SimCSE’s contrastive loss can be replaced with the batch-wise Barlow Twins loss.

[A] Gao et al., “SimCSE: Simple Contrastive Learning of Sentence Embeddings”, EMNLP 2021

Q5: A batch size of 256 is strongly detrimental for a contrastive approach. I suggest comparing with SimCLR in settings where the latter is known to excel.

We agree that contrastive methods have been shown to perform well at very large batch sizes. We see this as a negative of such approaches, as it increases the computational requirements of running such methods. Based on Figure 4, LFR does well at reasonably small batch sizes. However, we take your point, so for the new CIFAR10 experiments in the general comment we did increase the batch sizes to 512.

S3: It would be interesting to see how joint training behaves.

We did experiment with joint training, with an ablation on Kvasir shown in Figure 4. This ablation helps to justify why we used an EM-like algorithm rather than the simpler joint training idea.

We are very pleased to see that you enjoyed our paper, and will address the few comments you had.

W1: While LFR does not aim to improve performance on datasets like CIFAR10, it would be useful to know if LFR is better/worse than optimized augmentations, such as those used in SimCLR.

This was a common sentiment amongst reviewers, so we have provided discussion and experimental results on CIFAR10 in the general comment. In summary, LFR (using no augmentations) does not outperform SimCLR (using the full suite of augmentations) on natural image datasets. The same can be said for several domain-agnostic approaches including DIET and DACL.

W2.1: It would be interesting to perform ablations with e.g. MSE or Contrastive losses.

Since multiple reviewers brought up this point, we have addressed it in the general rebuttal by running experiments using LFR with InfoNCE instead of Barlow Twins. In addition to the arguments for our design decision in the paper, our batch-wise Barlow Twins loss did give a slight improvement over InfoNCE for downstream performance.

W2.2: It would also be interesting to see the transferability of the trained models across different datasets.

We originally did not include transfer learning experiments because they are not straightforward in the tabular and time-series domain. However, since it was a common request, we did find a suitable setting from prior literature and applied LFR to it. The results are discussed in the general comment.

Q1: I would be grateful if the authors could comment a bit on the choice of the number of projectors K=6 and batch size B=256. I would like to know if they have tried higher values for these two hyperparameters (especially for the number of projectors K).

As you noted, we performed ablations on Kvasir for both number of projectors and batch size in Figure 4 of Section 4.6. The largest values we used were $K=8$, and $B=512$. Already at these levels there was a decrease in final performance, so we have not extended it to larger values of $K$. Generally, our findings are that the projector selection approach from Section 3.3 is adequate for ensuring enough diversity in a small number of projectors that $K$ does not need to be large (which would add some cost to training, specifically from the $K$ predictor heads that try to match the $K$ projectors).

Thank you for your constructive review, we are happy to address each of your points.

W1 Discussion/comparison with other domain-agnostic methods seem to be not enough:

While we used three approaches to domain-agnostic SSRL (along with five strong domain-specific baselines), the methods you point out are also useful and we will cite them with discussion. Random Quantization (RQ) [Wu et al. 2023] has received attention with its recent publication at ICCV 2023 in October, which post-dates our submission to ICLR. Nevertheless, this is a helpful suggestion, so we re-ran SimCLR and SimSiam using RQ augmentations on three datasets, one from each modality. However, we found i-Mix [Lee et al.] to be very similar to the more recent DIET [Balestriero 2023], and given that it was shown to be outperformed by RQ [Wu et al. 2023], we believe our addition of a RQ baseline sufficiently covers these alternatives.

The results for SimCLR, SimSiam, and LFR are reproduced from Table 1, with new results added for RQ augmentations. Note that for the Income baseline we restated the SCARF result, since it is similar to SimCLR with random feature corruptions as the augmentation. RQ appears beneficial on HAR (Time Series), but is not necessarily helpful on Income (Tabular) and Kvasir (Image). We note that when Wu et al. applied RQ to image data, they combined it with more traditional image augmentations, as using RQ alone did not produce great performance (Tables 3, 4 in [Wu et al. 2023]). Ultimately, LFR still outperformed this baseline.

W2 Datasets are relatively small and not commonly used for benchmarking machine learning models.

We respectfully disagree on this point. Table 3 in Appendix C.1 summarizes all the datasets we used, which includes MIMIC-III at over 2.5M training examples, and HEPMASS at 7M. Compared with common datasets like CIFAR-10, these datasets are very large scale. The smaller datasets, HAR and Epilepsy, were chosen because they have previously been used for benchmarking time-series SSRL methods (e.g. Eledele et al. 2021), while Income and Theorem have been used for tabular benchmarking (e.g. Hajiramenzanali et al. 2022). In fact these datasets are commonly used: the paper introducing MIMIC-III has over 6000 citations [A], Kvasir has over 1000 combined [Pogorelov et al. 2017] [B], HAR over 2000 [Anguita et al. 2013], Epilepsy over 3000 [Andrzejak 2001], and Income over 2000 [Kohavi 1996]. As a side note, we did explicitly call out that Kvasir was a medical image dataset (not natural image) in Table 2 and Section 4.1.

You also mentioned two domain-agnostic works with more suggestions for other datasets to consider. [Lee et al.] uses image, speech, and tabular datasets, while [Wu et al.] experiment on images, point clouds, and audio. As can be seen, there is a lack of consistency in which modalities are tested for domain-agnostic work, and unfortunately we are not able to introduce several new modalities in the discussion period. However, to partially accommodate your requests, we have repeated our experimental setup on CIFAR-10 as a representative natural image dataset with the results shown in the general comment. The DABS benchmark does appear useful, and we will investigate it for the future.

[A] Johnson et al., “MIMIC-III, a freely accessible critical care database”, Nature Scientific Data 2016. [B] Jha et al., “Kvasir-SEG: A Segmented Polyp Dataset”, International Conference on Multimedia Modeling 2020.

Natural Images: Throughout the literature on SSRL, natural images are a common testbed, so it is unsurprising that several reviewers were curious about the performance of LFR on a dataset like CIFAR-10 (ze2f, gtio, 6gYm, Eryk). Since our work is about domain-agnostic representation learning, mainly in cases where it is not clear how to usefully augment the data, we focused on tabular, time-series, and medical imaging applications. Natural images, being a domain where semantic-preserving augmentations are plentiful, were not the intended target of our work. Still, it is understandable why the reviewers are interested in this topic, so we repeated our benchmarking on the CIFAR-10 dataset.

For CIFAR-10 we used the same experimental settings as for Kvasir in our paper (see Appendix C). As a brief recap, each method used ResNet-18 encoders, and 4-layer MLP predictors. For LFR, the projectors were randomly initialized 4-layer CNNs. Based on Reviewer Eryk’s comment that baseline methods like SimCLR benefit from large batch sizes, we increased this to 512 from 256 as used for Kvasir. The table below shows that, as one might expect, SimSiam and SimCLR achieve the best performance out of our baselines thanks to their extensively engineered pipeline of image augmentations. Two domain-agnostic approaches that do not use image augmentations, DIET and our LFR, fall into a second tier, outpacing the third tier of DACL, Autoencoder, and Random Init. This result is in stark contrast to what we found on Kvasir, where all methods ended up in a tight range, with LFR at the top. It seems clear that the heavy augmentations of SimSiam and SimCLR really are specialized for natural images, and are not suitable for other domains within the image modality.

Computational Costs: Reviewers ze2f and 6gYm asked about comparisons of computational costs between the proposed LFR, and baseline methods, especially with regards to the initial cost to select diverse random projectors using the DPP algorithm which is NP-hard. In brief, our experiments across diverse datasets consistently reveal that LFR's training time is often comparable to, or even notably shorter than the contrastive SSRL methods. For example, on Kvasir training SimCLR took around 9 hours, while LFR completed in just 2 hours on the same machine. This assessment of training time considers both CPU and GPU usage.

The efficiency in LFR's training time can be attributed to two primary factors. First, existing SSRL methods rely on extensive augmentations that are performed on CPU and can lead to data loading bottlenecks. LFR has no such issue. CPU-based image augmentations account for a significant amount of SimCLR’s 9 hour training time on Kvasir. For datasets with less computationally intensive augmentations, training times tend to align more closely - on the HAR dataset, LFR concluded training in approximately 3 hours while SCARF required around 3.3 hours.

Second, multiple passes through predictor heads in LFR is not overly burdensome. Unlike contrastive frameworks that use double augmentation and encoder passes, LFR requires fewer expensive operations: one pass through each random projector before training, and one pass through the encoder followed by one pass through each small predictor during training. Given that encoder updates are the most expensive, LFR's training time often proves less than SimCLR.

Compared to the cost of training an encoder, the diversity selection procedure using the Fast Determinantal Point Process (Chen et al. 2018) is an almost negligible contribution to the total training time. To recap, to select $K$ diverse projectors we initialize $N$ small neural networks and for each we run one forward pass on a batch of training data of size $m$. Then, DPP is applied over matrices of size $m^2\times K$ constructed from the network outputs. As a concrete example, on the Income dataset training LFR for 100 epochs took 849.21s on our machine, while the DPP selection took 3.01s.

Thank you for providing CIFAR-10 results, but some numbers are far lower than those reported in other papers, so the results here are not so convincing to me.

Specifically, below are some comparison of replication here vs. original DACL paper:

SimCLR: 86.7 vs. 93.4

DACL: 42.4 vs. 81.3 (83.8 for DACL+)

Also, this observation is indirectly agreed with [Lee et al.], which reported the performance of a similar mixup-based method without image augmentations as 83.4 on CIFAR-10. This contradicts to your claim that LFR outperforms DACL. At this point, it seems the proposed LFR is not better than other DA methods for some datasets (at least CIFAR-10) under the same condition that no data augmentation is used.

(Edit: I just realized that they used ResNet-50 while you used ResNet-18, but still I don't think that explains the reason why the performance of DACL drops almost 40%p.)

Regarding the claim that "CPU-based image augmentations account for a significant amount of SimCLR’s 9 hour training time on Kvasir (while LFR completed in just 2 hours on the same machine)," it seems you did not properly parallelize data augmentation; data augmentation is generally not a bottleneck of training time for image classification, based on my personal experience on contrastive learning on image datasets. The size of images in Kvasir is only 100x80 as stated in the end of Section 4.1, so I think the claim that SimCLR takes a longer training time than your proposed method due to extensive data augmentation is not so valid.

The reason why your experimental results are not convincing is due to the lack of fair comparisons on the datasets already used in prior works. Hence, I suggest authors to try out those used in prior works like [Lee et al.] or [Wu et al.] including CIFAR-10, and make a fair comparison with them to claim the superiority of the work. (I found that [Wu et al.] is published in ICCV2023 but available since Dec 2022, so it was online for almost an year. I believe this has been exposed for enough time to be recognized if you properly surveyed related works. Or, at least [Lee et al.] published in ICLR2021 is old enough and should be a good reference.)

Also, similar to DACL or [Lee et al.], it is good to try out both (domain-specific) augmentations and the proposed method together and see if it results in further improvements, as there is no reason not to use data augmentations if we know they are effective.

Thanks for your quick response! Below I provide more comments, that would hopefully be helpful regardless of the acceptance of this paper.

We emphasize that natural image tasks are not the main point of our work ..

I think it is inevitable to incorporate the CV (with natural images) and NLP domains if you claim the proposed method works across "a wide range of representation learning tasks that span diverse modalities and real-world applications", as CV and NLP are two major real-world applications that deep learning has been successfully applied.

Rather, if you are not willing to incorporate them, then I think it is better to explicitly limit your focus/contribution (in the title/abstract/intro) to medical image and/or tabular domains, and explain why the proposed method is specifically useful for such domains (if possible).

Regarding the low performance of DACL, generally speaking, if you cannot replicate a prior work on your side, then you can implement your method on top of the prior work's codebase. If the computational cost matters, then you can simply replace the arch with ResNet-18 on the official DACL code, or just go with i-Mix, as it provides ResNet-18 results. It is my surprising that you missed i-Mix, as I can find it within top-5 results by googling one of your keywords "domain-agnostic representation learning"

Regarding your statement "We are trying to make a fair comparison ..., for example, using the same encoders and batch size across tasks," note that using the same hyperparameter does not mean that the comparison is fair. Hyperparameters should be optimized per method via a proper validation process. Rather, you can match some properties/conditions that you believe they are important for a fair comparison, e.g., the wall-clock training/inference time.

SimCLR uses two forward/backward passes per datapoint; LFR uses one pass. Both of these factors are relevant for the overall training time comparison. The comparison we mentioned shows that LFR is not worse in terms of computational costs, which is what some reviewers suspected.

If you concern two forward/backward passes, then you can remove the mirrored loss in SimCLR (AFAIK this is better than reducing the number of iterations by a half), or just use the momentum encoder like MoCo/BYOL instead. It would be helpful to compare methods in a 2D graph, where the x-axis is the wall-clock training time and the y-axis is the performance.

Your answer to “The reason why your experimental results are not convincing is due to the lack of fair comparisons on the datasets already used in prior works…”

Again, if you cannot afford with the heavy experimental setting of DACL, then you can try i-Mix with ResNet-18. I have not thought additional experiments on datasets used in i-Mix or RQ can be done within the rebuttal period, so that's one of the main reason why my rating is not positive though I feel the proposed method is interesting.

It is not straightforward to merge an augmentation pipeline into our proposed method.

Is there a reason why you cannot apply data augmentation in your framework? Data augmentation is generally independent to the model architecture. Or, have you observed that naively applying data augmentation does not improve the performance of the proposed method, like masked autoencoders? I think that's fine, and providing the information about such fact might be useful to audiences.