Poly-View Contrastive Learning

Thank you for taking the time to read our work. We are happy you found the paper well-written and that you appreciate the benefits of being able to trade off batch size against views as well as the additional insights we provide about self-supervised learning algorithm design.

Please find below our comments on the raised issues and questions:

W1/Q1: Although there are prior works showing that multiplicity improves generalization and convergence of neural networks, it lacks rigorous theory on the relation between contrastive learnability and the number of views on each sample. I wonder how strong the theory on multiplicity can be.

Two-view MI objectives found in prior works on contrastive learning are a proxy to the InfoMax objective for representation learning. Our theory shows that one need not limit themselves to two-view MI to achieve InfoMax, but rather many other alternatives exist that use more positive views. Our paper studies One-vs-Rest MI, multiplicity being a controllable parameter and two-view being the case when the multiplicity $M=2$, yielding the following theoretical and empirical results:

1. Using multiple views in a pairwise comparison as in Multi-Crop, while reducing the variance of the MI estimator, does not result in a better bound for InfoMax (Proposition 2.1 and 2.2).
2. New contrastive losses that include multiple positive views, such as Arithmetic and Geometric PVC provide a tighter lower-bound on InfoMax (Theorem 2.3).
3. Sufficient statistics with minimal assumption also introduce a way of including more positive views into account. For a specific choice of sufficient statistics, we prove a connection between sufficient statistics and One-vs-Rest MI (Theorem 2.4) which shows that sufficient statistics also benefit from the above properties of One-vs-Rest MI.
4. Using synthetic and real-world (ImageNet1k) datasets, these theorems were validated in practical settings.

Please provide clarification regarding definitions of:

• Strength of multiplicity theories, and
• Contrastive learnability

in the case where we have not addressed your concerns.

Q2: Does there exist a threshold on the number of views, such that once it exceeds the threshold, more views do not help?

In the Growing Batch case (Figure 3, top row), one should always maximize the multiplicity $M$ as this provides the tightest bound on InfoMax (Theorem 2.3, 2.4).

In the Fixed Batch case where the total number of views $V=K\times M$ is held constant as $M$ increases, the answer is more interesting. Prompted by your comment, in Appendix E.5 we investigate this scenario in more detail with a simplified form of Geometric PVC, and are able to prove there does exist an optimal multiplicity $M^*$. Thank you for prompting us with this comment.

Thank you again for the time you took reading our work and providing the questions above. The additions to the work regarding Fixed Batch view multiplicity in particular have improved the work's overall completeness.

Thank you for taking the time to read and review our work. We are happy you appreciated the theoretical analysis of view multiplicity and the empirical justifications of the theoretical analysis.

Please find below our comments on the raised issues and questions:

W1.1: What is the loss for PVC?

Minimizing the PVC losses maximizes the MI lower bounds of Theorem 2.2. Based on your comment we have clarified the corresponding equations, making it clear what the loss is. Additionally, the full pseudo-code for all loss computations is provided in Appendix F.2.5, allowing any reader to follow the loss computation line by line and reproduce our results.

W1.2: How does M=2 link to SimCLR?

In Appendix E.2 we show how $M=2$ gives SimCLR for PVC. We have now added an equivalent derivation for sufficient statistics.

W2.1: Empirical evidence lacks suitable interpretation and linkage to the significance of the theorems. For significance, I mean how we can use the theorem takeaways to practically improve the SSL algorithms?

The PVC and Sufficient Statistic algorithms are a direct consequence of Theorems 2.2 and 2.4 respectively. These theorems tell you how to construct a Poly-View loss function. Without the theorems, one can use heuristics and develop Multi-Crop, which Poly-View methods outperform in all of our experiments.

In the synthetic case (Section 3.1) we observe experiments directly following theorems from Section 2. For example, increasing multiplicity $M$ increases One-vs-Rest MI as derived in Section 2.3.1 on data processing inequality. We also observe One-vs-Rest MI empirically converges to $\mathcal I(x_\alpha; c)$ which, as discussed in Section 2.3.1, is the upper bound for One-vs-Rest MI.

The ImageNet1k experiments follow similarly. Geometric and Sufficient Statistics with higher multiplicity $M$ provide tighter MI bounds and a better proxy for InfoMax, resulting in improved performance when holding other hyperparameters fixed (Figure 3a).

W.2: I expect to see the evidence on larger dataset with mainstream architecture such as ResNet and ViT/transformer with longer epochs

Please see Figure 3a for training and evaluation on ImageNet1k dataset, with a ResNet50 architecture trained for up to 1024 epochs.

W.3: It is unclear to me if the multiplicity of views simply benefits from more equivalent epochs (in the experiments) or whether the exposure to the number of data has been constrained to be exactly the same between the poly-view contrastive learning and other baselines

We were also concerned about this, and chose to focus on it in Section 3.2.

For Figure 3b, we train $M=2$ SimCLR models and $M=8$ Poly-View for 64, 128, 256, 512 and 1024 epochs. We do this either by holding the total number of views fixed (Fixed Batch. bottom row) which means we drop the batch size $B$ with increasing $M$, or allow total views to increase (Growing Batch, top row).

1. To answer whether it is the total number of samples seen by the model, we can look at the leftmost column of Figure 3a. We see that Poly-View methods always outperform SimCLR.
2. To answer whether it is the total number of updates seen by the model, we can look at the center column of Figure 3a. We see that Poly-View methods always outperform SimCLR.
3. To answer whether it is the total compute spent on the model, we can look at the right column of Figure 3a. Poly-View methods outperform SimCLR only in the Fixed Batch case, but not in the Growing Batch.

If effective epochs corresponds to total steps then in the above scenario 2 we see that for the same amount of effective epochs, Poly-View methods still outperform non-Poly-View methods.

We have also added a fourth way of comparing models in Appendix F.2.4 where we show the Total FLOPs used to produce each model. The conclusion is consistent with comparisons based on Relative Compute.

Please let us know if anything else is still not clear.

Dear Reviewer P4Q6,

We thank you again for taking the time to read and review our work. We really appreciated your comments and questions that prompted a number of clarifications and missing section links to be included in the paper, which has improved the overall readability of the paper. In particular, the section discussing the Geometric and Arithmetic PVC loss definitions is now much clearer, thank you.

Please let us know if you have any issues remaining with the work.

Best,

The Authors

Thank you for taking the time to read our work. We are happy you found the paper easy to read and you appreciate the benefits of tighter MI bounds, lower batch size, and reduced computational cost enabled by the Poly-View objectives, as well as the additional insight we provide about Multi-Crop.

Please find below our comments on the raised issues and questions:

W1: Exponential family assumption is strong

Our representations are given by a neural network and are therefore finite-dimensional (2048 in the ResNet50 case), typical in self-supervised learning. As the sufficient statistics is a map to the representation space, according to the Fisher-Darmois-Koopman-Pitman (FDKP) theorem, for smooth nowhere vanishing probability densities, a finite-dimensional sufficient statistic exists if and only if the densities are from an exponential family (https://ieeexplore.ieee.org/document/4048925). I.e. the sufficient statistics are distributed according to the exponential family because they map to finite-dimensional representations.

The assumption of finite-dimensional representations is not strong in the context of self-supervised learning. The exponential families that follow from this assumption are widely used in machine learning due to the availability of efficient methods as well as their tractability and interpretability. Many traditional distributions are in the exponential family, e.g. normal, exponential, log-normal, gamma, chi-squared, beta, Dirichlet, Bernoulli, categorical, Poisson, geometric, inverse Gaussian, von Mises, and von Mises-Fisher distributions. Finally, in self-supervised learning, the exponential families have appeared in many recent works, e.g. https://arxiv.org/abs/2102.08850, https://arxiv.org/abs/2203.07004, and https://proceedings.mlr.press/v89/hyvarinen19a.html.

We have reworded assumption 3 to make it clear it is a consequence of modeling representations using a neural network, and not an assumption. Thank you for highlighting this to us.

W2: The experiments do not show the superiority of Poly-View contrastive learning

The goal of our work is to understand how view multiplicity should be incorporated into contrastive learning from first principles and to investigate in a controlled ImageNet1k setting if there is any benefit to doing so in this way. Achieving state-of-the-art performance is not a goal of our work.

We found:

• Increasing multiplicity of Poly-View methods always gives better performance than the fixed multiplicity method SimCLR (Fig 3a top row first two columns, Fig 3b)
• Poly-View methods enable more compute-efficient design space than fixed multiplicity methods (Fig 3a bottom row final column).

We hope this helps clarify the purpose of our work and the benefits of a Poly-View approach which may be incorporated with orthogonal methods for increasing performance, e.g. increasing model size. We will continue to improve the clarity of the paper on this particular point. Please let us know if there is anything else you are looking for.

W3/Q1: Computation time is not evaluated by real time

In Figure 3 we evaluate computation time using:

• Total training epochs
• Total updates
• Relative Compute

These metrics capture different notions of time, as choosing only one or two of these metrics paints an incomplete picture. E.g., displaying only total updates does not take into account the practical consideration that a single update for one setting may require more computation than a single update for another, which would be captured by also including Relative Compute.

The three metrics above were also chosen as they are ML framework agnostic and hardware agnostic. Wall-clock time on the other hand can vary according to CUDA version, GPU model/compute capability network interconnect (e.g. infiniband/EFA availability). This makes wall-clock time difficult to use reproducibly.

Based on your comment, we have included total FLOPs, another popular, reproducible measure of computation time, in Appendix F.2.4, to complete the picture. We have also added a comment that, assuming an optimal hardware setting and fully-parallel implementation, total updates is proportional to wall-clock time, which we hope addresses your primary concern. Thank you again for your comment.

Dear Reviewer AhEA,

We thank you again for your thoughtful comments and questions. The resulting changes have increased the completeness and readability of the work.

We are still working to produce the requested fine-tuning results and will post them upon completion.

Please let us know if you have any other issues remaining with the work.

Best,

The Authors

Dear Reviewer AhEA,

We are happy to let you know that the majority of our fine-tuning transfer evaluations are complete and have been added to Appendix F.2.2.

For equivalent hyperparameters, we compared SimCLR against Growing Batch and Shrinking Batch variations of Geometric PVC. We did this for both small and large amounts of pre-training epochs, then fine tuning on a suite of natural tasks.

On the majority of natural tasks, Geometric PVC significantly outperforms SimCLR, and for the remaining tasks they are statistically equivalent according to standard statistical tests. This demonstrates the utility of Poly-view methods on real tasks, addressing the outstanding W4/Q1. Thank you again for suggesting this investigation to us.

Please let us know if you have any other issues remaining with the work.

Best,

The Authors

Thank you for taking the time to read our work. We are happy you appreciate the empirical and theoretical validation of the Poly-View contrastive methods, their practical benefits as measured by efficiency, and recognize the broad applicability for improving self-supervised learning.

Please find below our answer to your raised question.

Q1: I just have one suggestion: maybe you can comment (or leave for future work) about how other self-supervised learning can fit in your framework, or how your idea can be extended to other SSL methods like BYOL etc.

Our primary contributions use the frameworks of information theory and sufficient statistics to investigate what is possible in the presence of a view multiplicity $M>2$, and derive the different Poly-View objectives from first principles.

It is possible to incorporate multiplicity $M>2$ into a distillation setup like BYOL (https://arxiv.org/abs/2006.07733). For example, DINOv1 (https://arxiv.org/abs/2104.14294), which shares many algorithmic parts of BYOL, benefits a lot from using the pair-wise Multi-Crop task that we describe in Section 2.2 and Appendix D (although in DINOv1, there is more than one augmentation policy).

One option for extending distillation methods like BYOL/DINOv1 from Multi-Crop to Poly-View type tasks in a One-vs-Rest sense is to have the EMA teacher produce $M-1$ logits, which are aggregated into a single logit (similar to the sufficient statistics choice for $Q$ in Equation 24) for producing the target pseudo-label distribution. The gradient-based student could then be updated based on its predictions from the held-out view, and this procedure aggregated over all the view hold-outs.

Such a procedure is very interesting and definitely worth future consideration. The core difference between the distillation procedure above and the Poly-View contrastive methods is that the large-view limit of Poly-View contrastive methods is provably a proxy for InfoMax (Section 2.1 and Equation 27). There may be a way to obtain theoretical guarantees for the large-view distillation methods (using for example tools from https://arxiv.org/abs/2307.10907), and could prove an interesting future direction for investigation.

We have included a discussion regarding extensions of other self-supervised learning methods in Appendix H. Again, we appreciate the time you took to read our work and share your valuable comments and enthusiasm.