EMP-SSL: Towards Self-Supervised Learning in One Training Epoch

• The overall novelty of the paper is unclear. This work uses Bag-of-Features (BoF) through a fixed-patch approach, and regularization via TCR. Both these strategies have already been explored, as the authors mention. As is, the paper lacks clarity with respect to its novelty.

• The comparison in "epochs" seems unfair to other methods. Efficiency in SSL should not be measured in terms of epochs, but rather in time/compute or some surrogate metric such as number of patches processed. I would also include a comparison in terms of GPU memory required per method.

  • Similarly, I could not find any discussion about the efficiency of EMP-SSL at test time.

• Although the method is compared to several other SOTA methods, the datasets used are rather small to extract general conclusions about SSL. Larger datasets (>= ImageNet-1K) would be required for that, specially because of the use of larger resolution images which might affect (positively or negatively, to be seen) the patching and BoF approach.

• I personally think the language in the paper could be improved. There are several informal expressions that could benefit from some rewriting. I also found several typos, I suggest the authors giving a careful read to polish those aspects, since language is an important aspect in the reading of a scientific paper.

• In most of the experiments, the accuracy number of the proposed method is much lower than that of the baseline methods. Although I understand that the main point of the paper is to say that this method can converge faster, I feel this conclusion is irrelevant when the final accuracy is off by a lot. For example, in Table 1, the accuracy of the proposed methods is at least 5% lower on any dataset, and even 20% lower on the most realistic one (ImageNet-100). This makes this table kind of useless in my opinion.
• The authors don't provide a clean table comparing the wall-clock time spent to achieve the same SOTA accuracy using their method versus other baseline methods. This seems to me the most important thing to include, because the definition of 'epoch' in this paper is kind of irrelevant given that many more augmentations are used in each epoch in their method. Only the wall-clock time seems to be really reasonable metric to show that this method is faster.
• Despite trying to include a little bit of result on ImageNet-100, most results (especially those that are used to argue the wall-clock time improvement) are based on CIFAR which is too small in modern deep learning.

Given the current empirical results (which is not completely convincing to me) and the nature of the method (not very noval given that it's just using more patches wih VicReg), I don't think it's ready to be published yet unless more convincing details are included.

While the authors present an innovative method, it appears that the paper lacks experiments to analyze the effectiveness of the method, and the results are still not convincing. Here are a few specific points of concern:

• The literature suggests that SSL methods benefit from larger batch sizes (for instance, SimCLR uses 4096). However, the batch sizes considered by the authors range from 0 to 32 (including the ablation study in the supplementary material). This significant variation in the number of batch iterations and weight updates makes a fair comparison with other methods difficult. Conducting experiments with a consistent batch size would help determine if these methods, with a large number of updates, can achieve reasonable performance. Consequently, the results in tables 1 and 2 lack credibility.
• The out-of-domain data experiment presented in table 5, involving cifar10 and cifar100, lacks depth. While the left part of the table might be suitable for testing the model's out-of-domain capability, the right part lacks usefulness. A more insightful approach would involve testing the model on a completely different dataset to better assess its transferability.
• The paper overlooks experiments related to downstream tasks, which are crucial in SSL for evaluating the quality of features learned by the model.
• The ablations and experiments conducted on cifar10 lack informativeness. A larger dataset could have been utilized for more robust analysis.
• The paper's narration needs enhancement, particularly on page 2, where method details are repetitively presented, leading to confusing notations.
• The paper lacks a hyperparameter analysis, including sensitivity analysis concerning parameters such as learning rate and lambda in the TCR loss function.

• W1: The so-called one epoch is a little tricky, since EMP-SSL use 20-200 views for one image, which is far more than conventional methods have used. EMP-SSL will use at least 10 times of the data as much as conventional SSL methods use in one epoch.
• W2: I do not see results with ResNet-50 network, which may be a more standard setting for self-supervised learning. Some conclusions reached with smaller networks may not be generalized to larger networks.
• W3: While I approve of the note on time efficiency, the comparison of time efficiency is not intuitive. I still wonder how many epochs can be run using conventional methods like SimCLR in the time of running 1 epoch of EMP-SSL.
• W4: The authors mentioned co-occurrence in their motivation of the methods. However, I have no idea what the co-occurrence is represented here and how this technical term is defined.
• W5: The paper may lack some ablation study on the choice of the loss function. Given 200 views of one image, I believe there are more than one ways to design the loss function and the regularizer, i.e. InfoNCE loss.