Using Contrastive Learning with Generative Similarity to Learn Spaces that Capture Human Inductive Biases

• The first major weakness is in the methodology and the underlying message of the paper Quality. The paper focusses on demonstrating that data augmentations in contrastive learning are not sufficient to capture human inductive biases compared to direct supervision. This is demonstrated by comparing the proposed solution against a self-supervised contrastive learning strategy, SimCLR. However, such comparison is not fair for two reasons: 1. It is unfair to have a supervised vs. unsupervised comparison. The authors should provide the same amount of supervision to the baseline in order to demonstrate the benefits of their proposed objective. 2. The authors should provide a comparison with existing supervised deep metric learning strategies, as ultimately this is the problem considered in the paper.
• It is not clear what is the underlying novelty of the paper compared to deep metric learning, as discussion and comparison is missing Novelty/Quality. At a more conceptual level, the novelty is rather limited and incremental. The proposed objective is essentially a traditional metric learning objective, as it boils down to have the following form $E_{x,x^+,x^-\sim p(X,X^+)p(X^-)}\{\ell(x,x^+,x^-)\}$. Is the novelty related to the way supervision is provided? The benefits should be discussed and supported through experiments.
• Important related work is missing Quality. First of all, discussion about the relation with deep metric learning should be provided. Moreover, there are recent works on the formal definition of generative similarity and Bayesian interpretations of self-supervised learning, see for instance [1,2]
• The experimental methodology lacks essential details, which undermines its soundness Quality/Soundness. Very limited information is provided about training, raising doubts about the general validity of the results, especially for the comparison with the baseline. For instance, on tieredImageNet which ResNet architecture has been used and why is training performed only on 10 epochs? This is not common practice in self-supervised learning. Authors should test their solution on traditional self-supervised learning benchmarks like ImageNet-1k to better support their claims against contrastive learning.

References
[1] Kim, Puthawala, Chul Ye, Sansone. (Deep) Generative Geodesics. In ICML 2024 Geometry-grounded Representation Learning and Generative Modeling Workshop
[2] Sansone, Manhaeve. GEDI: GEnerative and DIscriminative Training for Self-Supervised Learning. arXiv 2022

• Sec 3.2, 3.3, and 3.4, the generative similarity objective aligns a lot with the goal of the tasks for the model. Optimizing for it surely can be considered an inductive bias in the models, but it is consequently not surprising to observe that the models outperform the baseline without this objective, and even similar objectives such as SimCLR (as the similarity in SimCLR is object-wise instead of category based). Have the authors considered investigating some emergent properties due to this objective, such: as training on the most regular shapes and testing on the least regular in Sec. 3.2; testing on another style of drawing in Sec. 3.3; or evaluating the robustness of the model in Sec. 3.4. These suggestions are examples of ways to evaluate whether this objective brings any new properties to the models outside of the scope of the loss function. They do not need to be necessarily tested by these types of experiments would strengthen the claims of the authors.

• The authors mention that this objective can be considered as an additional step towards better alignment with humans. But the link with humans is very tangential in this article, except in Sec 3.2. To validate the role of such an objective in the human brain, have the authors considered specific types of behavioral studies or neural datasets they could use to further validate the human-like properties of the learned representations?

The experiments offer little support that the central innovation---generative similarity---has practical value. Standard triplet losses are used for all but the experiment about trying to replicate aspects of human visual processing (2). Experiments 3 and 4 involve relevant representation learning tasks, but they compare contrastive learning where triplets are relevant to the task at hand to triplets formed via standard image augmentations (SimCLR). It’s not surprising that the former is more successful: if you are able to group the data into positives and negatives that relate to the task you want to solve, better performance isn’t coming from a helpful inductive bias, it’s coming from supervision.

Tying similarity to a specific generative model makes the method’s practical utility highly dubious, and the experiments do not shed light on how accurate the assumed generative model must be for the similarity to yield useful representations. If generative similarity is only useful when one already has a perfect generative model for the data, I don’t see why it’d provide additional value.

• The framework relies on manually provided inductive biases. This raises concerns about scalability and feasibility for more complex or real-world applications.
• The method is confined to the triplet loss only, which limits its usability for other settings.
• I feel the usefulness of the method is not well shown overall. The paper only demonstrates that the network trained on generative similarity performs better than just a single baseline of SimCLR, in a simple setting. If the performance is the goal, I think there are a lot of ways to improve it for the given task. Additionally, while the advantages of instilling inductive biases into machine learning models, such as improved accuracy, few-shot learning, interpretability, and robustness, were mentioned in the intro, only performance gain in very simple settings is shown in the paper.

• For the experiment on the tieredImageNet dataset, the authors assume access to abstract categories for each image. However, the comparison is done with SimCLR with random augmentations, where no such information is utilized. SimCLR uses positive samples as random augmentations of the anchor image since it doesn’t have access to image labels or secondary information that can help with the sampling procedure. It might be more fair to compare with say SupCon (Supervised Contrastive Learning) which uses ground truth image labels to improve the representation quality.

  Moreover, for the bottom level of abstraction when sampling triplets, the authors construct the positive image as another sample from the same category and same class as the anchor, and the negative image as sampled from the same category but independent class. This bears similarities to hard negative sampling in contrastive learning (https://arxiv.org/abs/2010.04592) which again improves the representation. A more fair comparison should be made with such baselines, assuming access to the same amount of secondary information.

  Furthermore, the authors merely present the accuracy of both approaches for Abstract Category classification. How does generative similarity affect the quality of representation in terms of performance on downstream tasks (linear probe), nearest neighbors (KNN), etc? This analysis is specifically important to show how human-based inductive biases help in learning a richer representation.

• While there are interesting experiments demonstrating proof of concept and the key idea, there is no comparison with existing baselines (such as https://proceedings.neurips.cc/paper_files/paper/2023/file/9febda1c8344cc5f2d51713964864e93-Paper-Conference.pdf and many more). It is unclear what is the quantitative impact of this learning paradigm and how it compares theoretically or empirically to these existing models.

• I find it very interesting how the authors are able to propose for instilling inductive biases in neural networks without the need to collect human similarity judgments at scale. However, their approach assumes access to a hierarchical generative model of the data, which is a significant and potentially limiting assumption. Obtaining such data at scale can be just as resource-intensive, and in many cases, characterizing a hierarchical generative model is challenging.