Contrastive Self-Supervised Learning As Neural Manifold Packing

We thank the reviewer for the careful review. Below, we provide our detailed, point‑by‑point responses.

• "The main weakness lies ..." : In Section 5 we deliberately track sub‑manifold sizes and nearest‑neighbour counts because these metrics are intrinsic to CLAMP’s augmented submanifold loss; contrastive baselines like SimCLR, BYOL, and MoCo do not explicitly form or optimize over sub‑manifolds, so these measurements simply don’t map onto their training dynamics. For Sections 6 and 7, we instead draw on the published Brain‑Score benchmarks and eigenspectrum exponents reported in the MMCR paper (NeurIPS 2023: 24103–24128). We have extracted the relevant numbers from MMCR and included them alongside our own results to enable a fair comparison, showing that CLAMP’s manifold‑packing framework yields distinct gains in neuroscientific alignment and geometric uniformity over these established methods.

• "There is extensive literature ..." : Thanks for referring to these two works. We will add the sentences below to the related work section in the finalized version.

  Hard‑negative methods draw the negative samples from the tailored Boltzmann-like distributions to sharpen sample‑wise contrasts.

  Collapse‑proof non‑contrastive methods preserve feature diversity and avoid collapse by implicitly clustering representations and enforcing augmentation invariance without negative samples.

We thank the reviewer for time and efforts. Below, we provide our detailed, point‑by‑point responses.

• " The "with only local operations" argument ... " : Thanks for pointing this out. The reviewer is correct that the optimization of any active pair of overlapped manifolds changes the encoder and, as a result, all representations are modified. We will clarify what we mean by "with only local operations" and will test explicitly the effect on the centroids of inactive manifolds due to an operation on the centroids of two active manifolds.

• "It is not clear which ... " : Thank you for raising this point. The projection head implemented as an MLP (whose outputs we refer to as embeddings) acts as an effective scaffold during contrastive training. In both SimCLR (arXiv:2002.05709) and MoCo v2 (arXiv:2003.04297), inserting this projection head during training and removing it at inference time consistently improves downstream classification accuracy compared to using no projection head at all. Following their success, subsequent methods such as BYOL and Barlow Twins have adopted the same architecture. Here, we also incorporate an MLP projection head to enhance our model’s training performance. In our revised ablation study, we will test the effect of not including the projection head.

• "The difference between MMCR ..." : MMCR seeks to maximize the “manifold capacity” of a set of N low dimensional manifolds. This should be understood in the sense of “perceptron capacity”, i.e. MMCR seeks to improve the relative arrangement of the manifolds so that a linear classifier can shatter as many random assignments of binary labels (1 or 0) to the N manifolds, in certain technical limits. So MMCR ignores the fact that the N manifolds belong to multiple distinct classes, and instead probes the problem as a linear binary classification of low-dimensional manifolds carrying one of two labels. So, MMCR is not explicitly trying to pack manifolds, but it achieves manifold separation because it tends to favor higher perceptron capacity (note however that this is quite indirect and less interpretable than the clear physical picture provided by CLAMP). CLAMP instead directly addresses the N-ary problem of packing and nonlinearly classifying N distinct manifolds in a simple and physically interpretable framework. We will clarify what we mean by local interactions as stated above, and will test explicitly the effect on the centroids of inactive manifolds due to an operation on the centroids of two active manifolds.

• " The results on brain-score ..." : We appreciate the reviewer’s suggestions about our Brain‑Score (Section 6). Our intent in these sections was not to claim that CLAMP perfectly mirrors every stage of visual cortex, but rather to probe whether a repulsive objective, with no explicit biological constraints, nonetheless gives rise to any of the statistical hallmarks we observe in real neural populations and shows the potential of implementing CLAMP by biologically plausible learning rules. The limitations of Brain-Score have been discussed in section 8. We will modify section 8 to show more detailed limitations about the Brain-Score.

Brain-Score’s linear mapping between deep nets output and experimental activity may overlook non-linear relationships between model and neural representation. It also does not guarantee that CLAMP implements the same computations or dynamics as cortex.

• "The argument on a similar ..." : We appreciate the reviewer’s feedback on Section 7. As noted by Stringer et al. (Nature, 571(7765):361–365, 2019), smooth neural manifolds are expected to exhibit a power-law exponent greater than 1 across a range of experiments involving different input types and varying fractions of sampled neurons. In this context, our analysis demonstrates that the representations learned by CLAMP exhibit a similar power-law behavior. While we agree that multiple controllable parameters can influence the exponent, we view our findings as a meaningful extension of Stringer et al.’s test, providing an additional perspective on the structure of learned representations.

• "Do CLAMP have meta-parameters ..." : Thanks for the question. CLAMP’s packing loss is self-balancing: by minimizing the objective it either shrinks submanifold sizes (improving alignment) or pushes them farther apart (improving uniformity), with no extra meta-parameter required to trade off these effects. In other words, the alignment–uniformity balance emerges implicitly from the packing loss itself. As we demonstrate in Section 4.1:Linear Evaluation, the CLAMP packing loss not only simplifies the formulation but also outperforms the original alignment-uniformity method.

We thank the reviewer for the constructive feedback. Below, we provide our detailed, point‑by‑point responses.

• "Given that the central ..." : we present preliminary results on object detection using a Faster R-CNN head with a C4 backbone (pretrained on ImageNet-1K with 4 views for 100 epochs), fine-tuned on VOC2007+2012 training dataset and tested on VOC2007 test dataset. Comparative metrics for other methods are sourced from the MMCR paper (arXiv:2303.03307). | VOC2007+12| mAP | AP50 | AP75| | --------------- | :-------------: | ---------------:|---------------:| | Barlow Twins | 53.1 | 80.9 | 57.7 | | SimCLR | 54.4 | 81.6 | 61.0 | | MoCo v2 | 54.7 | 81.7 | 60.2 | | BYOL | 55.6 | 82.3 | 62.0 | | MMCR | 54.6 | 81.9 | 60.0 | | CLAMP | 55.6 | 82.2 | 61.5 | *Higher mAP/ AP50/AP75 shows better performance. Overall our results show that CLAMP is competitive with the state of the art. We will further improve object detection by training on the larger COCO dataset and through hyperparameter tuning. We will include the new results in the revised version of the manuscript.

• "The paper's claims hinge ..." : For perturbative Gaussian noise as the augmentation transformation where the input is x and the transformation is $x+\zeta$ with $\zeta \sim \mathcal{N}(0, \Sigma)$, the augmentation submanifold is provably elliptical. The resulting augmented embedding follows a Gaussian distribution: $\mathcal{N}\left(g(f(x)), J_x(g(f(x))) \ \Sigma \ J_x(g(f(x)))^\top \right) + O(\zeta ^3)$, where $J_x( \cdot)$​ denotes the Jacobian matix. $f(\cdot)$ and $g(\cdot)$ denote the encoder and projector network. For more general augmentations, the resulting submanifold is not necessarily elliptical. In such cases, we can approximate the augmentation submanifold with a sufficiently large enclosing sphere, as done in CLAMP. However, this approximation may lose fine geometric details and underutilize the space’s packing capability. Conversely, modeling the submanifold with more accurate geometries demands significantly more augmentations to estimate, on the order of $O(D)∼100-1000$ in high-dimensional spaces, which is computationally costly untenable at scale. Thus, there is a trade-off between augmentation cost and packing efficiency.We have mentioned this point in Section 8, line 351.

• "In the training dynamics (Figure 3b) ..." : A zero-volume submanifold occurs when an image and all its augmentations collapse to the same representation, reflecting perfect transformation invariance. While this invariance can aid image classification, it may discard the fine details needed for other downstream tasks. CLAMP’s packing loss naturally prevents such collapse: radii only contract while two submanifolds overlap, and once they become merely tangent, the shrinkage gradient disappears. As a result, each radius settles at a nonzero “margin” value instead of vanishing.

• "The performance of many ... " : Thank you for raising this point. Here we show the linear evaluation results of pretrained models on ImageNet-1K for 100 epochs with n_views=4 and rs=8.5. | Batch sizes | 256 | 512 | 1024 | | --------------- | :-------------: | ---------------:|---------------:| | Linear evaluation | 69.0 | 69.4 | 69.1 |

The training performance depends weakly on the batch size. Such weak dependency has also been observed for MMCR.We will further improve the ablation study by including this table and additional results that we are currently generating for different numbers of views.

We thank the reviewer for the time and efforts on reviewing our work, which have helped improve our manuscript. Below, please find our detailed responses.

• " While the geometric structure..." : CLAMP’s repulsive packing loss isn’t directly grounded on a specific synaptic learning rule. However, its main idea, encouraging competition and decorrelation among sub‑manifolds may be analogous to anti‑Hebbian plasticity. Bridging CLAMP’s high‑level objective with a concrete, biologically‑plausible learning model remains a very important direction for future work. We have mentioned this point in Section 8, line 343-346.

• "Evaluation: The evaluation ..." : Thanks for raising this interesting point. Our t‑SNE visualization reveals that categorical labels naturally segregate suggesting an emergent structure in the embedding space. By analogy, we hypothesize that CLAMP’s representations will likewise organize into hierarchical clusters (subcategorical clusters within parent categories). We will discuss this hypothesis and seek supporting evidence for it on ImageNet in the revised version of the manuscript.

• " While packing helps ..." : The representation learned by CLAMP demonstrates its ability to distinguish between different categories, indicating strong classification capabilities. This property is beneficial for a variety of downstream tasks. Accordingly, fine-tuning self-supervised models on object detection and segmentation tasks often yields high performance. Here, we present preliminary results on object detection using a Faster R-CNN head with a C4 backbone (pretrained on ImageNet-1K with 4 views for 100 epochs), fine-tuned on VOC2007+2012 training dataset and tested on VOC2007 test dataset. Comparative metrics for other methods are sourced from the MMCR paper (arXiv:2303.03307). | VOC2007+12| mAP | AP50 | AP75| | --------------- | :-------------: | ---------------:|---------------:| | Barlow Twins | 53.1 | 80.9 | 57.7 | | SimCLR | 54.4 | 81.6 | 61.0 | | MoCo v2 | 54.7 | 81.7 | 60.2 | | BYOL | 55.6 | 82.3 | 62.0 | | MMCR | 54.6 | 81.9 | 60.0 | | CLAMP | 55.6 | 82.2 | 61.5 | *Higher mAP/ AP50/AP75 shows better performance. Overall our results show that CLAMP is competitive with the state of the art. We will further improve object detection by training on the larger COCO dataset and through hyperparameter tuning. We will include the new results in the revised version of the manuscript.

• "The practical benefit ... " : It is interesting to apply this type of self-supervised loss function to other tasks including multimodal tasks. We will mention this possibility in the discussion and reserve a detailed benchmark on these large-scale datasets for future work, commensurably with the available computational resources.

• "What practical benefits ..." : CLAMP uses a single repulsive potential whose parameters have clear geometric meaning, making the training dynamics tractable. The manifold‑packing view immediately suggests richer objectives such as adaptive radius schedules analogous to the Lubachevsky-Stilliger packing algorithm. We will include this discussion on the possibility of deriving new physically-motivated objectives in Section 8.