Probabilistic Self-supervised Representation Learning via Scoring Rules Minimization

We thank the reviewer for the positive acknowledgment and valuable comments. We refer to General Response 1 for the impact of our proposed method on mitigating collapsing representation.

Dear Reviewer JrU7,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript. As the deadline for the discussion nears, we would like to ask you to participate. We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks, Authors

We thank the reviewer for the valuable comments and suggested experiments that highlighted our contributions. Please refer to General Responses 1 and 2 for W3.1 and W3.3 respectively.

Q1. ... The most important difference between BYOL and ProSMin

• We employ a similar architecture to DINO (ViT-S) for both the encoder and the projection head, incorporating three linear layers with Gelu activation. Notably, our projection head does not include batch normalization. We then add a prediction layer similar to BYOL. Even without a batch normalization layer, we use a higher representation of 16k compared to 4096 for BYOL. However, we have shown that the performance does not suffer drastically at lower dimensions. Then, we have two layers for mean value and standard deviation. Our loss function is the proper scoring rule loss based on energy score L1 loss, and BYOL uses MSE loss. So we use a prediction layer like BYOL in terms of architecture, but we don't have batch normalization and use a different encoder and projection head, and we use another loss function based on a probabilistic approach. So, our method is different from BYOL, and we also showed that batch normalization and the prediction layer are not necessary components to prevent the collapse scenario, unlike BYOL.

Q2., W1., W2. Suggested improvements and typos

• We thank the reviewer for pointing out the typos and we have updated our manuscript and incorporated these suggestions.

W3. ...Is the author's solution inspired by either of the two causes or did it address either of the two factors?

• Thank you for asking this question, we address collapsing representation without overparametrization and heavy data augmentation instead we push the boundaries of self-supervised learning by embracing the richness of probabilistic models. By adopting a probabilistic framework, our proposed method encourages the neural network to assimilate a comprehensive distribution of representations, rather than being constrained to a singular, fixed representation.

W3.2. Performance comparison with various baseline methods for the same feature dimensions.

• We have the experiments for different embedding sizes in Figure 3-d for 100 epochs, and we also add a new result with the same embedding size as DINO. We summarize the knn results in the following Table: | Methods | embedding in the loss |knn| |----------:|------:| -----:| | iBOT | 8192 | 71.4| | Our method | 8000 |71.2| | DINO | 65536 | 69.7| | Our method | 65536 |69.9|

Dear Reviewer pLHM,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript. As the deadline for the discussion nears, we would like to ask you to participate. We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks, Authors

We thank reviewer QnX8 for the useful comments and suggested improvements. Please refer to General Response 1 for the first part of Q3.

Q1. How do you ensure $q_{\theta}$ is not an identity network w.r.t $\mu$ with $\sigma$ = 0 since the dimensionality of $z_{\theta}$ and $t_{\theta}$

• Thanks for pointing out this issue. To avoid this case, the $q_{\theta}$ contains two linear layers with GELU as nonlinearity and uses ReLU nonlinearity between $q_{\theta}$ and $\mu$, so $q_{\theta}$ is a nonlinear function between $t_{\theta}$ and $\mu$ in case $\sigma$ is zero.

Q2. Since your scoring function is a variation of L2 loss between $z_i$ and $z_{\xi}$, can you show that the variational encoding actually makes a difference by perturbing $t_{\theta}$ by a Gaussian noise of ${\sigma}= \epsilon$

• The hyperparameter ${\lambda}$ is the determining factor for the effect of the perturbation. Thus, a lower value of ${\lambda}$ shows a more significant effect on the second-term portion of the loss, which is dominated by the injected Gaussian noise. Figure 3-a shows the effect of ${\lambda}$, lower ${\lambda}$ shows better performance compared to a higher value of ${\lambda}$. This means that injecting Gaussian noise improves the performance of the model. However, we adjust ${\lambda}$ to keep the loss function positive, which is essential for the proper scoring rule. Therefore, there is a limit to the lower value of ${\lambda}$, depending on the case.

Q3.2. ...This is important because the authors claim that strong augmentations distort the underlying distribution, hence promoting the collapse of the representations learned.

• The effect of augmentation on collapsing representation has been discussed extensively for contrastive learning by Jing et al.[1].

Reference: [1] Jing, L., Vincent, P., LeCun, Y., & Tian, Y. (2022). Understanding dimensional collapse in contrastive self-supervised learning. ICLR.

Dear Reviewer QnX8,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript. As the deadline for the discussion nears, we would like to ask you to participate. We just wonder whether there is any further concern and hope to have a chance to respond before the discussion phase ends.

Many thanks, Authors